System for cooling hard disk drives using vapor momentum driven by boiling of dielectric liquid

ABSTRACT

A system for cooling hard disk drives (HDDs) includes: an enclosure having a lower volume within which a cooling liquid is heated to a boiling point temperature to cause some of the cooling liquid to evaporate into a plume of rising vapor; a HDD cooling area with at least one HDD placed in the direct path of the rising vapor, which cools the at least one HDD during functional operation of the at least one HDD; and a heat source that dissipates heat into the lower volume of the enclosure, sufficient to heat the cooling liquid to the boiling point temperature. The system can also include a condenser located above both the HDD cooling area. A substantial portion of the rising vapor that passes through the HDD cooling area and cools the at least one HDD is condensed back into liquid phase on contact with the condenser.

BACKGROUND

1. Technical Field

The present disclosure generally relates to cooling systems forelectronic devices and in particular to cooling systems for hard diskdrives (HDDs). Still more particularly, embodiments of the presentdisclosure relate to techniques for cooling hard disk drives (HDDs)using vapor momentum.

2. Description of the Related Art

As the value and use of information continue to increase, individualsand businesses seek additional ways to process and store information.One option available to users is information handling systems. Aninformation handling system generally processes, compiles, stores,and/or communicates information or data for business, personal, or otherpurposes thereby allowing users to take advantage of the value of theinformation. Because technology and information handling needs andrequirements vary between different users or applications, informationhandling systems may also vary regarding what information is handled,how the information is handled, how much information is processed,stored, or communicated, and how quickly and efficiently the informationmay be processed, stored, or communicated. The variations in informationhandling systems allow for information handling systems to be general orconfigured for a specific user or specific use such as financialtransaction processing, airline reservations, enterprise data storage,or global communications. In addition, information handling systems mayinclude a variety of hardware and software components that may beconfigured to process, store, and communicate information and mayinclude one or more computer systems, data storage systems, andnetworking systems.

Large scale server systems are examples of information handling systems.These servers can perform significant workloads and generate a largeamount of heat during their operation. A significant portion of the heatis generated by the operation of hard disk drives (HDDs), which can belocally (that is, internally or externally) connected on a same chassisas the processors and other units of the server. Due in part to thelarge amount of heat generated, these servers are typically rack mountedand cooled via internal fans and/or fans attached to the back of therack or elsewhere within the server ecosystem. As the need for access togreater and greater processing and storage resources continues toexpand, the density of server systems (i.e., the amount of processingpower and/or storage placed on a single server, the number of serversplaced in a single rack, and/or the number of servers and or racksdeployed on a single server farm, continue to increase. With the desirefor increasing processing or storage density in these server systems,the thermal challenges that result continue to be one of the biggestobstacles. Conventional fan based cooling systems require large amountsof power, and the cost of power required to drive high flow fansincreases exponentially with the increase in server densities. What istherefore needed is an efficient, low power usage system and method forcooling the HDDs and other components of the server, while allowing forthe desired increased processing and/or storage densities on theindividual server chassis.

BRIEF SUMMARY

Disclosed are a system, a method, and an information handling systemconfigured as a multi-phase heat transfer immersion cooling tank forenabling cooling of hard disk drives (HDDs) by exposing the HDD to vapormomentum driven by boiling of a dielectric liquid below the location ofthe exposed HDDs. Also disclosed are a system and method and aninformation handling system that enables increase in processing and/orstorage density on a server chassis by locating the HDDs of the serverat an elevated location in an immersion cooling tank in which theprocessing components of the server chassis are submerged. Theprocessing, memory and other components of the physical servers, exceptthe HDDs, are directly cooled by submerging the physical servers in avolatile dielectric liquid within a multi-phase heat transfer immersiontank, while the HDDs are located above the dielectric liquid for vaporcooling. The localized cooling of these HDDs enables high HDD densitieslocalized to the server environment.

According to a more general aspect of the disclosure, a system forcooling HDDs includes: an enclosure having a lower volume within which aliquid can be heated; and a HDD cooling area in which at least one HDDcan be placed during operation of the HDDs, wherein the HDD cooling areais located at a first distance above a top surface layer of the liquidwithin the enclosure and is in a direct path of vapor rising off theliquid. The system further includes a heat source that dissipates heatinto the lower volume of the enclosure sufficient to heat the liquidwithin the lower volume to a boiling point temperature at which at leasta portion of the liquid evaporates generating a mass/plume of risingvapor. The rising mass of vapor flows/moves rapidly upwards through theHDD cooling area and across one or more surfaces of the at least oneHDD. The moving vapor cools the at least one HDD via convection as thevapor comes into contact with and moves across the one or more surfacesof the at least one HDD.

In addition to the above components, one or more embodiments providethat the system also includes: a condenser located above both the HDDcooling area and the at least one HDD; and a condensed liquid returnsystem, a portion of which is located above the HDD cooling area. Thecondenser includes a condensation surface and a condensation fluid thatflows proximate to the condensation surface and which maintains thecondensation surface at a lower temperature than a condensation point ofthe rising vapor. Accordingly, in the described embodiments, a majorityof the rising vapor that passes through the HDD cooling area and coolsthe at least one HDD is condensed back into liquid phase on contact withthe condensation surface. According to one or more embodiments, thecondenser comprises at least one extended length of tubular pipingrunning from the outside of the enclosure to the inside of the enclosureto create the condensation surface, and the condensation fluid flowsthrough the interior bore of the tubular pipe from an external fluidsource, such as a reservoir. The condensation fluid is at a temperaturethat is less than the condensation temperature of the rising vapor.

According to yet another embodiment, the condensation surface is angledto at least one side relative to a horizontal plane to cause the liquidgenerated from the condensation of the rising vapor to flow off towardsat least one fluid collection and return area. The fluid collection andreturn area includes a return channel that directs the collected fluidtowards the lower volume of fluid without coming into contact with theat least one HDD.

In one or more embodiments, the condensed liquid return system comprisesfluid collection and return areas on an interior side of the enclosurethat is located away from the HDDs so as to prevent the condensed fluidfrom being able to fall back onto the at least one HDD). The condensedliquid return system collects the condensed liquid as the liquid dropsfrom the condensation surface, due to gravity, following condensation ofthe rising vapor. The condensed liquid return system is located abovethe at least one HDD so as to protect the HDD from having any of thecondensed liquid fall on the operating HDD. The condensed liquid returnsystem includes a second portion that provides a return conduit/channelwhich extends from the first portion down into the enclosure past theHDD cooling area. Accordingly, the condensed fluid is returned to thelower volume of the enclosure, and contact with the at least one HDD bythe condensed liquid is avoided. In one embodiment, the HDD cooling areais designed with at least one area at which no HDDs are located andthrough which the return channel runs.

In one embodiment, the HDD cooling area includes a rigid, mesh typestructure extended across the inner perimeter of the enclosure toprovide a porous platform on which the HDDs are placed. In anotherembodiment, the HDD cooling area includes a set of holding clips andsleeves designed to receive one or more of the HDDs and hold the HDDs inplace within the HDD cooling area. The HDDs are then held in placewithin the HDD cooling area based on the orientation and/or location ofthe holding component. The HDDs are spaced apart from each other toincrease surface area contact with the rising vapor. The system alsoincludes one or more cables connected to the HDDs and which enable (a)data communication exchange with one or more connected processingdevices that utilize the at least one HDD and (b) receipt of power tothe at least one HDD. The cables are extended out of the HDD coolingarea and are rated to operate at a temperature that is at least amaximum of (a) an ambient exterior temperature of the HDDs and (b) atemperature of the rising vapor.

According to one or more embodiments, the enclosure is a server rackimmersion cooling tank and the immersion cooling tank includes an uppervolume housing the condenser, HDD cooling space, liquid return system,and a cover that can be designed to provide an air-tight seal whenclosed on top of the enclosure. Also, according to one aspect, thecooling tank includes a server rack. With this configuration, the heatsource is at least one operating processing device, such as a server,which is submerged in the cooling liquid and which dissipates heatduring operation of the server. Further, the liquid is a dielectricliquid that has high volatility and a low boiling point relative to thetemperature increase caused by the heat being generated by (ordissipated from) the server(s). Accordingly, during normal serveroperations, and/or while the server is powered on, the server dissipatesheat sufficient to raise the temperature of the dielectric liquid to theboiling point temperature of the liquid. The increased temperature thuscauses the vaporization of a portion of the dielectric liquid, and theevaporating dielectric liquid provides the rising vapor that cools theHDD.

According to one aspect, the disclosure provides an information handlingsystem having a multi-phase heat transfer immersion tank designed toenable cooling of hard disk drives (HDDs) located above a dielectricliquid within the tank. The immersion tank comprises a plurality of thecomponents and functional features described above.

In one or more embodiments, and alternate configuration of both thephysical server and the immersion cooling vessel is provided. Thisalternate configuration provides a new configuration of the informationhandling system, which is referred to as a vertical LCIS. The verticalLCIS is designed with (1) a motherboard generally aligned in ahorizontal position and having processing components and memory modulesattached thereto and (2) a vertically aligned backplane connected to andextending above the motherboard and on which one or more HDDs arephysically connected. The vertical LCIS is then placed within animmersion cooling drawer having a lower volume in which the motherboardsits within dielectric fluid, while the HDDs can be supported by thebackplane above the dielectric fluid. In alternate embodiments, the HDDsare not supported by the backplane, but rather by some other form ofsheet metal utilized to supplement or entirely carry the load of theHDDs. The immersion cooling drawer is then placed within an immersioncooling cabinet, which provides a sealed environment with a condenser atthe top of the cabinet to complete the heat transfer from the risingplume of dielectric vapor. The HDDs extend over the dielectric liquid,supported by the vertical backplane and are vapor cooled by the risingplume of dielectric vapor.

The above summary contains simplifications, generalizations andomissions of detail and is not intended as a comprehensive descriptionof the claimed subject matter but, rather, is intended to provide abrief overview of some of the functionality associated therewith. Othersystems, methods, functionality, features and advantages of the claimedsubject matter will be or will become apparent to one with skill in theart upon examination of the following figures and detailed writtendescription.

BRIEF DESCRIPTION OF THE DRAWINGS

The description of the illustrative embodiments can be read inconjunction with the accompanying figures. It will be appreciated thatfor simplicity and clarity of illustration, elements illustrated in thefigures have not necessarily been drawn to scale. For example, thedimensions of some of the elements are exaggerated relative to otherelements. Embodiments incorporating teachings of the present disclosureare shown and described with respect to the figures presented herein, inwhich:

FIG. 1 illustrates one embodiment of an example information handlingsystem with internal processing components and externally located harddisk drives (HDDs), according to one aspect of the disclosure;

FIG. 2 is a schematic diagram of an information handling systemconfigured as a server with an upper section of hard disk drives whenoriented to stand on a back section of the chassis, according to one ormore embodiments;

FIG. 3 is a schematic diagram of another example of an informationhandling system configured as a server with a set of externallocally-connected hard disk drives, according to one or moreembodiments;

FIG. 4 is a schematic diagram illustrating a first example internalstructure of a two-phase vaporization-condensation immersion coolingtank with a dielectric cooling liquid that operates to provide coolingfor processing components of one or more servers partially immersed inthe cooling liquid with the server's hard disk drives located above thecooling liquid in the path of rising dielectric vapor, according to oneor more embodiments;

FIG. 5 is a schematic diagram illustrating a second example internalstructure of a two-phase vaporization-condensation immersion coolingtank configured to support cooling of the example server of FIG. 3, withexternal locally-connected hard disk drives suspended above thedielectric cooling liquid in the path of the rising dielectric vapor,according to one or more embodiments;

FIG. 6 provides various views of a rubber-based grommet that can beutilized within the immersion cooling tank to allow for insertion ofpower connectors and data cabling, while preventing leakage of thecooling liquid or vapor from the immersion cooling tank, according toone embodiment;

FIG. 7 is a flow chart illustrating one example of a general method forcooling HDDs within an immersion cooling vessel and controllingoperating conditions within the immersion cooling tank via feedbackcontrol, according to one or more embodiments;

FIG. 8 illustrates a three dimensional view of an example immersioncooling tank such as presented by FIGS. 4 and 5, configured withmultiple condenser sub-units to provide cooling of multiple partiallyimmersed servers placed within a server rack, in accordance with one ormore embodiments;

FIG. 9 is a facility exchange diagram showing the application of thevaporization-condensation cooling system to a larger heat exchangeenvironment, according to one embodiment;

FIG. 10 illustrates a plurality of sequentially linked heat exchangeloops utilized for cooling a target space with cycling of working fluidsutilizing multiple heat exchanges and associated working fluids intandem with a working fluid reservoir, according to one or moreembodiments;

FIG. 11 illustrates an example of a system providing multiplecondensation-vaporization and vaporization-condensation cycles ofworking fluid loops to effect a heat transfer and/or cooling of a targetspace, according to one embodiment;

FIG. 12 is an example immersion tank for cooling one or more partiallyimmersed servers utilizing one of a plurality of circulating channelsfor condensation of the rising dielectric vapor and return of thecondensate to the lower volume of the tank, according to one embodiment;

FIG. 13 illustrates examples of methods for gravitationally controllinga level of cooling liquid in a first immersion cooling tank utilizing apipe connection to a secondary volume of cooling liquid, in accordancewith one or more embodiments;

FIG. 14 is a flow chart illustrating a method of controlling thedielectric liquid level within an immersion cooling tank, in accordancewith one or more embodiments;

FIG. 15 is a block diagram illustrating an aerial view of example daisychaining of multiple immersion cooling tanks to maintain cooling fluidequilibrium across the multiple tanks within a data center, inaccordance with one or more embodiments;

FIG. 16 illustrates an example vapor pressure control sub-systeminvolving deployment of a bellows expansion lid within a tank cover ofan immersion cooling tank having multiple adjacent bellows within thetank cover, according to one or more embodiments;

FIGS. 17A-17B provides two additional views of an example vapor pressurecontrol sub-system including multiple adjacent bellows within the tankcover of a single immersion cooling tank, in accordance with oneembodiment;

FIG. 18 illustrates example detection and feedback control mechanismsdeployed within an immersion cooling tank and which enable control ofvarious operational conditions internal to the immersion cooling tank,including vapor pressure reduction, during operation of the tank as acooling vessel, in accordance with one or more embodiments;

FIG. 19 illustrates an example proportional-integral-derivative (PID)algorithm that can be utilized within a pressure control system for animmersion cooling tank, in accordance with one embodiment;

FIG. 20 is a flow chart illustrating a method of controlling pressurebuild up within an immersion cooling tank, in accordance with one ormore embodiments;

FIG. 21A illustrates an example vertically-oriented server configurationin which a vapor deflector is provided to direct rising vapor bubblesaway from upper components that are submerged in cooling liquid,according to one embodiment;

FIG. 21B illustrates a second example of a vertically-oriented serverconfiguration in which multiple vapor deflectors are provided to directrising vapor bubbles away from upper portions of a single componentand/or upper components that are submerged in cooling liquid, accordingto one embodiment;

FIG. 21C illustrates an example of a high heat dissipating componentthat includes both deflector fins as well as heat fins attached to thesurface of the component, according to one or more embodiments;

FIG. 22 is a three dimension view of an example motherboard of avertically-oriented liquid and vapor cooled immersion server (vLVCIS)having processing components and memory modules located on opposingsurfaces of a shared motherboard, in accordance with one or moreembodiments;

FIG. 23 presents a three dimensional view of an example vLVCIS withstorage devices located vertically above the other functional componentsthat are embedded on the opposing sides of the motherboard of FIG. 22,according to one or more embodiments;

FIG. 24 is a three dimensional schematic of an immersion server drawerhaving multiple side-by-side vLVCISes located therein, according to oneor more embodiments;

FIG. 25 presents a three dimensional illustration of an exampleimmersion server drawer cabinet providing a vapor condensation chamberfor housing multiple immersion server drawers, according to one or moreembodiments; and

FIG. 26 is a three dimensional schematic of a stand-alone,self-contained, immersion tank data center (ITDC) including an immersiontank with one or more servers partially immersed in a dielectric liquidto enable cooling of the server components during operation, accordingto one or more embodiments.

DETAILED DESCRIPTION

The present disclosure provides illustrative embodiments of variousaspects of and/or different configurations and implementations of one ormore systems, methods, and multi-phase heat transfer immersion coolingvessels that enable direct cooling of information handling systems, suchas servers, by submerging at least a portion of the physical informationhandling systems in a dielectric liquid within a multi-phase heattransfer immersion cooling vessel.

The disclosure generally includes a plurality of different aspects andmultiple different embodiments, and each aspect along with theassociated embodiments are described in detail below within one of thetitled Sections A-K. A first aspect of the general disclosure, presentedin Section A, provides examples of an information handling system and oftwo different servers configured and/or oriented for use within arack-based immersion cooling system. Section B, which describes thesecond aspect of the general disclosure, introduces the design andoperation of an example immersion cooling tank, including thedescription of certain functional characteristics of the immersioncooling tank. A third aspect of the general disclosure is presented inSection C, which provides techniques for cooling hard disk drives (HDDs)of a server by exposing the HDDs to rising vapor from a boilingdielectric liquid in which the processing components are submerged forcooling during processing operations of the server. The fourth aspect ofthe general disclosure is presented in Section D, which introduces theinnovative concept of submerging PDUs in order to prevent electricalarching and improve the efficiency of the PDUs when utilized within animmersion cooling vessel. A fifth aspect of the general disclosure ispresented in Section E, which provides a unique design and functionalityof the condenser with multiple rotatable condenser sub-units. A sixthaspect of the general disclosure is presented in Section F, whichdescribes a methodology for cooling a target space and/or a device thatinvolves stepped sequencing of multiple heat exchangers (or condensers)with different working fluids. A seventh aspect of the generaldisclosure is presented in Section G and includes a uniqueinterconnection among multiple immersion tanks to allow cooling liquidlevels to be gravitationally equalized across the multipleinterconnected immersion tanks. An eight aspect of the generaldisclosure is presented in Section H, which provides a series oftechniques for controlling and/or mitigating the buildup of pressurewithin the tank, as well as other control functions, in order tomaintain the integrity of the tank (e.g., from high pressure vaporleakage, etc.). Sections I and J introduce several novel server designaspects including a vapor deflector for isolating processors, animmersion server, immersion server drawer, and an immersion serverdrawer-based cabinet, all designed to facilitate immersion-based liquidcooling of the processors and memory modules of the immersion server andvapor cooling of the HDDs. Finally, an eleventh aspect of the generaldisclosure is presented in Section K, which provides a Stand-aloneImmersion Tank Data Center (SITDC) with self-contained cooling.

Additional functional aspects of the general disclosure are presentedthroughout the description of one or more of the sections. It isappreciated that the description of certain functional aspects of thedisclosure within a particular section (rather than in a differentsection, for example) is presented in that section solely to simplifythe order of presentation of information and does not and/or is notintended to limit the specific disclosure content to discreteimplementation within only each identified section. Rather, the sectionswork together to provide a description of a single generally innovativeconcept with multiple different aspects and/or embodiment correspondingthereto.

In the following detailed description of exemplary embodiments of thedisclosure, specific exemplary embodiments in which the disclosure maybe practiced are described in sufficient detail to enable those skilledin the art to practice the disclosed embodiments. For example, specificdetails such as specific method orders, structures, elements, andconnections have been presented herein. However, it is to be understoodthat the specific details presented need not be utilized to practiceembodiments of the present disclosure. It is also to be understood thatother embodiments may be utilized and that logical, architectural,programmatic, mechanical, electrical and other changes may be madewithout departing from general scope of the disclosure. The followingdetailed description is, therefore, not to be taken in a limiting sense,and the scope of the present disclosure is defined by the appendedclaims and equivalents thereof.

References within the specification to “one embodiment,” “anembodiment,” “embodiments”, or “one or more embodiments” are intended toindicate that a particular feature, structure, or characteristicdescribed in connection with the embodiment is included in at least oneembodiment of the present disclosure. The appearance of such phrases invarious places within the specification are not necessarily allreferring to the same embodiment, nor are separate or alternativeembodiments mutually exclusive of other embodiments. Further, variousfeatures are described which may be exhibited by some embodiments andnot by others. Similarly, various requirements are described which maybe requirements for some embodiments but not for other embodiments.

The terminology used herein is for the purpose of describing particularembodiments only and is not intended to be limiting of the disclosure.As used herein, the singular forms “a”, “an” and “the” are intended toinclude the plural forms as well, unless the context clearly indicatesotherwise. It will be further understood that the terms “comprises”and/or “comprising,” when used in this specification, specify thepresence of stated features, integers, steps, operations, elements,and/or components, but do not preclude the presence or addition of oneor more other features, integers, steps, operations, elements,components, and/or groups thereof. Moreover, the use of the terms first,second, etc. do not denote any order or importance, but rather the termsfirst, second, etc. are used to distinguish one element from another.

It is understood that the use of specific component, device and/orparameter names and/or corresponding acronyms thereof, such as those ofthe executing utility, logic, and/or firmware described herein, are forexample only and not meant to imply any limitations on the describedembodiments. The embodiments may thus be described with differentnomenclature and/or terminology utilized to describe the components,devices, parameters, methods and/or functions herein, withoutlimitation. References to any specific protocol or proprietary name indescribing one or more elements, features or concepts of the embodimentsare provided solely as examples of one implementation, and suchreferences do not limit the extension of the claimed embodiments toembodiments in which different element, feature, protocol, or conceptnames are utilized. Thus, each term utilized herein is to be given itsbroadest interpretation given the context in which that term isutilized.

Within the descriptions of the different views of the figures, the useof the same reference numerals and/or symbols in different drawingsindicates similar or identical items, and similar elements can beprovided similar names and reference numerals throughout the figure(s).The specific identifiers/names and reference numerals assigned to theelements are provided solely to aid in the description and are not meantto imply any limitations (structural or functional or otherwise) on thedescribed embodiments. In the illustration of the various embodiments,two different figures can be provided that have overlaps and/orsimilarities in the components within the two figures (e.g., FIGS. 2 and3, and FIGS. 4 and 5). In such instances, the descriptions of thesefigures can be presented together with associated similar referencenumerals separated by commas and/or a slash. Some components that arenot expected to be different from one implementation to the other areprovided the same reference numerals within the figures, to simplify thedescriptions of the figures.

Those of ordinary skill in the art will appreciate that the hardwarecomponents and basic configuration depicted in any of the figuresillustrated by the drawings and described herein may vary. For example,the illustrative components within information handling system 100 (FIG.1), example server 200, 300 (FIGS. 2 and 3), immersion server 2100, 2300(FIGS. 21 and 23), and/or immersion cooling vessels/tanks 400, 500, 800(FIGS. 4-5, 8), and other devices and systems are not intended to beexhaustive, but rather be representative of and highlight componentsthat can be utilized to implement aspects of the present disclosure. Forexample, other devices/components may be used in addition to or in placeof the hardware depicted. The depicted examples do not convey or implyany architectural or other limitations with respect to the presentlydescribed embodiments and/or the general disclosure.

A. Information Handling System and Server Configuration or Orientationfor Rack-Based Immersion Cooling

Turning now to the figures, FIG. 1 illustrates a block diagramrepresentation of an example information handling system (IHS) 100, withwhich one or more of the described features of the various embodimentsof the disclosure can be advantageously utilized. For purposes of thisdisclosure, an information handling system, such as IHS 100 and/orserver 200 (FIG. 2) or server 300 (FIG. 3) or immersion server 2200(FIG. 22) may include any instrumentality or aggregate ofinstrumentalities operable to compute, classify, process, transmit,receive, retrieve, originate, switch, store, display, manifest, detect,record, reproduce, handle, or utilize any form of information,intelligence, or data for business, scientific, control, or otherpurposes. For example, an information handling system may be a handhelddevice, personal computer, a server, a network storage device, or anyother suitable device and may vary in size, shape, performance,functionality, and price. It is however appreciated that the informationhandling systems of the present disclosure are described as beingprimarily rack-based server systems. The information handling system mayinclude random access memory (RAM), one or more processing resourcessuch as a central processing unit (CPU) or hardware or software controllogic, ROM, and/or other types of nonvolatile memory. Additionalcomponents of the information handling system may include one or moredisk drives, one or more network ports for communicating with externaldevices as well as various input and output (I/O) devices, such as akeyboard, a mouse, and a video display. The information handling systemmay also include one or more buses operable to transmit communicationsbetween the various hardware components.

Referring specifically to FIG. 1, example IHS 100 includes processingcomponents 150 (referred to as “internal” components) that arephysically located on the device chassis and other functional componentslocated externally to the processing components 150 (referred to as“external” components). Included within processing components 150 are aplurality of processor(s) 105 coupled to system memory 110 via systeminterconnect 115. System interconnect 115 can be interchangeablyreferred to as a system bus, in one or more embodiments. Also providedwithin the processing components 150 and coupled to system interconnect115 is one or more storage controller(s) 120, to which is coupled localinternal storage 122 and local external storage 125. Storagecontroller(s) 120 controls data storage operations on nonvolatilestorage, of which local internal storage 122 and local external storage125 are shown.

As presented herein, local internal storage 122 can be solid statedrives (SSDs), while local, external storage 125 are hard disk drives(HDDs). It is appreciated that the presentation of local internalstorage 122 separate from local external storage 125 within the presentdisclosure extends from the concept that certain types of non-volatilestorage and particularly SSDs can be operated effectively whilesubmerged in a fluid medium along with the processing components 150,while other types of non-volatile storage, such as HDDs, may operatebetter if not submerged in a liquid medium. However, in one embodiment,the local internal storage 122 can also comprise or be solely HDDs,while the local external storage 125 can include SSDs. Also, in analternate embodiment, information handling system 100 can include onlylocal internal storage 122, regardless of the type (i.e., SSD, HDD,etc.) of storage. As presented herein, the terminology “local external”indicates that the storage 125 is an integral part of the informationhandling system 100, but connected via a data cable at a short distancefrom the processing components 150 (i.e., off the device chassis) ratherthan being directly connected on the device chassis via the systeminterconnect 115. Also, in at least one embodiment, the local externalstorage 125 is located near enough to the main processing components 150that there is minimal additional latency detected between accessinglocal internal storage 122 and accessing local external storage 125.

In one embodiment, the non-volatile storage (122/125) provides storageof application programs, software, firmware, and one or more sets ofdata for use by processors 105 during one or more data processingoperations. As shown, system memory 110 can include therein a pluralityof modules, including firmware (F/W) 112, basic input/output system(BIOS) 114, operating system (O/S) 116, and application(s) 118. Systemmemory 110 further includes one or more feedback control modules 170. Inone embodiment in which the immersion cooling tank is a self-controlled,intelligent device, program code of these feedback control modules 170can be executed by processor 105 and can configure the informationhandling system 100 to control one or more operating features of theimmersion cooling vessel and/or data center, as described in furtherdetail herein. The various software and/or firmware modules 170 havevarying functionality when their corresponding program code is executedby processor(s) 105 or other processing devices within IHS 100.

Within the above description and throughout the disclosure, referencesto processing components refers to components that execute tasksequences, while storage components refers to components utilized tosecure either the input or output of a processing event. Both processingand storage are needed in increasing densities across the computer andserver industry, in varying degrees. It is appreciated that in one ormore embodiments, the term “processing” can be defined to be theholistic inclusion of task execution and storage thereof.

IHS 100 further includes one or more input/output (I/O) controllers 130which support connection by, and processing of signals received from,one or more connected input device(s) 132, such as a keyboard, mouse,touch screen, or microphone. I/O controllers 130 also support connectionby, and forwarding of output signals to, one or more connected outputdevices 134, such as a monitor or display device or audio speaker(s).Additionally, in one or more embodiments, one or more device interfaces136, such as an optical reader, a universal serial bus (USB), a cardreader, Personal Computer Memory Card International Association (PCMCIA)slot, and/or a high-definition multimedia interface (HDMI), can beassociated with IHS 100. Device interface(s) 136 can be utilized toenable data to be read from or stored to corresponding removal storagedevice(s) 138, such as a compact disk (CD), digital video disk (DVD),flash drive, or flash memory card. In one or more embodiments, deviceinterface(s) 136 can also provide an integration point (e.g., a USB orserial port) for connecting other device(s), such as an external HDD, toIHS 100. In such implementation, device interfaces 136 can furtherinclude General Purpose I/O interfaces such as I²C, SMBus, andperipheral component interconnect (PCI) buses. In the illustrativeembodiment, each of input devices 132, output devices 134, and deviceinterfaces 136 with removable storage devices 138 are illustrated asexternal to processing components 150, while I/O controller(s) 130reside within processing components 150.

In one embodiment, IHS 100 includes a power controller 140, which iscoupled to a power distribution unit (PDU) 145 that is locatedexternally to IHS 100. PDU 145 provides a connection to the source ofelectrical power and power controller 140 provides and/or supportsdistribution of that electrical power within IHS 100 via a powerdistribution network (not shown). IHS 100 also comprises a networkinterface device (NID) 160, which is also included within processingcomponents 150. NID 160 enables IHS 100 to communicate and/or interfacewith other devices, services, and components that are located externalto IHS 100. These devices, services, and components can interface withIHS 100 via an external network, such as example network 165, using oneor more communication protocols. Network 165 can be a local areanetwork, wide area network, personal area network, and the like, and theconnection to and/or between network 165 and IHS 100 can be wired orwireless or a combination thereof. For purposes of discussion, network165 is indicated as a single collective component for simplicity.However, it is appreciated that network 165 can comprise one or moredirect connections to other devices as well as a more complex set ofinterconnections as can exist within a wide area network, such as theInternet.

With reference now to FIGS. 2 and 3, there are presented schematicdiagrams of respective information handling systems presented as twoconfigurations of a server. First server 200 and second server 300differ with respect to the location and connectivity of HDDs 125relative to the server chassis. First server 200 and second server 300(collectively referenced herein as server 200/300) include a casing 205within which the majority of the functional components are assembled.The casing 205 is designed to be able to insertably fit within a serverrack and allow connection to power cables and/or data cables and otherconnecting cables provided at or by the server rack. Casing 205 includesa pair of handles 207/307 for lifting and/or pulling server 200/300 whenthe entire server unit needs to be moved from one physical location toanother or in order to insert and/or remove server 200/300 from a serverrack (not shown). Located within casing 205 is a chassis or motherboard210 on which is embedded or attached the various processing componentsof server 200/300. As illustrated, these processing components include aplurality of processors (or processor integrated circuit chips) 105,storage controller 215 and memory modules 220. Server 200/300 can alsoinclude heat dissipation surfaces (i.e., heat sinks) co-located with theprocessing components. According to one aspect of the disclosure, one ormore embodiments provide that heat generated and dissipated from thesurface of the processors 105 and memory modules 220 are directlyabsorbed by a cooling fluid in which the processing components of server200/300 is submerged during operation. Thus, server 200/300 can bedesigned without heat sinks, in one or more embodiments. Casing 205 isdesigned with a porous sidewall 225 having a plurality of holes and/oropenings 230, 232. Thus, server 200/300 can be a standard server that isdesigned for operation in an air environment (i.e., outside of aliquid-based immersion tank) such that the server components can beair-cooled. However, as described herein, aspects of the disclosureutilizes these holes and/or openings 230, 232 to allow for a coolingfluid to easily flow into and around the internal structure and/orcomponents of server 200/300. The cooling fluid can thus surround thevarious processing components that dissipate heat and directly absorbthe dissipating heat by conduction and convection forces. These featuresare described in greater detail in the descriptions of FIGS. 4-5 and 8.

Server 200/300 are representative of any information handling systemthat can be partially or completely immersed or submerged in a coolingliquid to cool one or more of the functional components operatingthereon. As one related aspect of being an immersible or submersibleserver, server 200/300 is shown oriented in an upright or verticalposition, with the processing components at the bottom and the drivespace 235 for housing HDDs 125 located at the top. Notably, server 200does include on-chassis hard drive space(s) 235 within which HDDs 125are located, as an integral part of the server 200. In contrast, server300 is shown having HDDs 125 external to and not an integral part of theserver chassis 210. As indicated by the dashed line representations ofHDDs, one embodiment can allow for a standard server chassis 210 andcasing 105, wherein the HDDs 125 are removed and connected via cable 305to allow for positioning above the server casing 205, when second server300 is in a liquid cooled environment. Notably, however, drive space 335of second server 300 can accommodate a plurality of HDDs 125 to enablesecond server 300 to be reconfigured with HDDs 125 located within drivespace 335 when second server 300 is being air cooled.

In the illustrated configuration, the HDDs are referred to as locallyexternal HDDs. Thus, because second server 300 is designed for usewithin an immersion cooling tank, second server 300 is shown as beingconfigured with locally external HDDs 125, which are connected via cable305 to second server 300. Cable 305 can be a data cable, power cable,and/or a combination data and power cable. This configuration of secondsever 300 allows for a local external connection of one or more HDDs,separate from the other processing components of the device in order tobe able to submerge the processing devices in liquid for coolingthereof. The presented example of second server 300 illustrates that thelocation of the HDDs can be at a short distance away and apart from thechassis/motherboard 210 due to expected immersion of the chassis 210within the immersion cooling liquid. It is appreciated that theconfiguration of second server 300 can also be different, given the lackof on-chassis HDDs. The chassis 210 can, in one embodiment, be morecompact, eliminating the upper section 350 including the drive space 335altogether. In at least one embodiment, the configuration of firstserver 200 can also be different from a conventional server, byextending the middle portion 240 of the chassis between the processingcomponents and the drive spaces 235 to provide more separation of theHDDs 125 decreasing the likelihood of contact by the HDDs 125 with thecooling liquid. Other differences can also be provided to enable use ofserver 200/300 within an immersion cooling vessel. However, given thepossibility of these various differences, the configuration and/ordesign of example servers 200/300 presented herein are not intended toimply or convey any limitation with respect to the actual configurationand/or layout and/or type of server that is ultimately provided as theimmersion server that is cooled by submerging at least the processingcomponents within a cooling liquid that is provided in an immersioncooling tank, as described herein. FIGS. 22-25 and the descriptionsthereof in Section I present a different configuration of immersionservers that are specifically designed to be liquid cooled, withintegrated HDDs that are vapor cooled. However, for purposes of thegeneral concepts of the disclosure, reference shall be made to server200/300 to illustrate the application of the disclosed cooling conceptsto existing server designs and configurations.

Further, it is appreciated that the specific designs and configurationsof these servers (200/300) can in some instances affect the specificimplementation of the immersion cooling vessel utilized and/or theimmersion cooling features presented by the disclosure. However, thecore concepts of immersion cooling apply to the various possible typesof servers regardless of whether the servers include locally internalHDDs or locally external HDDs.

B. Immersion Cooling Tank Providing Cooling of Information HandlingSystems with Two-Phase Vaporization-Condensation Cooling Cycle

Turning now to FIGS. 4 and 5, there are presented an internal view oftwo different examples of immersion cooling tanks 400 and 500,respectively designed for insertion of first server 200 and secondserver 300. It is appreciated that the internal views provided ofimmersion cooling tank 200, 300 are transverse views running in thelateral direction of the illustrated server 200/300, such that the sideof the server chassis is shown (without functional components). Alateral view running parallel to the direction of server 200/300 ispresented by FIG. 8. Again, because of the similarities in the figures,the descriptions are presented together with associated similarreference numerals separated by commas and/or a slash.

Each immersion cooling tank 400/500 operates as a two-phasevaporization-condensation cooling vessel for cooling one or moreinformation handling systems and in particular server 200/300 accordingto the various methodologies described hereinafter. Immersion coolingtank 400/500 includes an enclosure 405 having an exterior casing with aninterior lower volume 402 within which cooling liquid 412 can bemaintained and heated to a boiling point temperature. The cooling liquid412 is generally located in the volume extending between the bottom ofthe enclosure 405 of cooling tank 400/500 up to the liquid surface 420,i.e., the topmost layer/level of cooling liquid 412. The interior ofenclosure 405 also includes an upper volume 404 extending from theliquid surface 420 up to a tank cover 480. In the illustrativeembodiments, tank cover 480 includes a handle 485 to allow for openingof the tank cover 480.

Also, according to one aspect, each immersion cooling tank 400/500includes a server rack 410/510 (illustrated with a side rail and a basestructure), which can be generally incorporated or placed within thesidewalls and base of the lower volume 402 of the tank enclosure 405,with at least one section extending below the liquid surface 420 ofimmersion cooling liquid 412. Within the lower volume 402 is shown abase portion and opposing side rails of the server rack 410/510. Serverrack 410/510 can respectively provide support for holding server 200/300in place while server 200/300 is submerged in cooling liquid 412. In oneembodiment, server rack 410/510 is built into immersion cooling tank400/500 or permanently attached to the interior sides of the side and/orbottom panels of casing 405/505. Server rack 410/510 can also beseparate from and placed into the existing immersion cooling tank400/500 following construction of the immersion cooling tank 400/500.Positioned within each server rack 410/510 is respective server 200/300which are shown extending laterally across immersion cooling tank400/500 from the first side rail (located on the left) to the opposingsecond side rail (located on the right).

Immersion server 200 is illustrated partially submerged below the liquidsurface 420, with the section of the chassis holding the HDDs 125 abovethe liquid surface 420 of the cooling liquid. In contrast, immersionserver 300 is illustrated with the chassis fully submerged below theliquid surface 420, with the locally external HDDs 125 above the liquidsurface 420 of the cooling liquid 412. In this version of immersioncooling tank 500, immersion server 300 can include handles 307 extendingfrom the cooling liquid 412 to allow for removal of the immersioncooling server 300 from the cooling liquid 412. The extension of handles307 above the liquid surface 420 enables server 300 to be pulled up outof the cooling liquid without contacting the cooling liquid 412. In oneor more embodiments, one or both of casing 405 and handles 207/307 ofrespective server 200/300 can be elongated to extend above the surfacelevel 420 of cooling liquid 412 within immersion cooling tank 400/500.Alternatively, the level of cooling liquid 412 can be measured such thatonly the heat-dissipating components on server 200 that are to be liquidcooled are immersed in cooling liquid 412.

According to one embodiment, extending from casing 405 of immersioncooling tank 400/500 is a pipe connector 495 with an associated shutoffor flow control valve 497. One or more of the functionality or use ofthese components are described in Section G, which is provided below. Asits simplest function, pipe connector 495 and flow control valve 497enable cooling liquid 412 to be drained from immersion cooling tank400/500, in order to facilitate tank maintenance, for example.

As shown in the figures, the HDDs 125 are located within a middleportion (or volume) of the immersion cooling tank 400/500 in which aplume of vapor is shown rising from the surface of the cooling liquid.This aspect of the design of the immersion cooling tank 400/500 andrespective servers 200/300 will be described in greater detail in thefollowing Section C.

The upper volume of the enclosure 405 includes at least one condenser460, which as provided by the inset includes a condensation surface 465and pipes 467 in which condensation fluid flows. The condensation fluidis maintained at a temperature that is lower than the condensationtemperature of the rising vapor 422. At the condenser 460, the risingvapor 422 is condensed back to cooling liquid or condensate 462 byreleasing latent heat to the condenser as the rising vapor 422 comesinto contact with the condensation tubes or surface in which thecondensation liquid flows. The condensation of the rising vapor 422occurs as the flowing, cooler condensation fluid within the condenserabsorbs the heat energy from the rising vapor 422, causing the risingvapor 422 to convert (i.e., undergo a phase change) from gas to liquidphase. The resulting converted/condensed cooling liquid 462 is thenreturned back to the lower volume of cooling liquid 412 in the bottom ofthe enclosure 405. In one embodiment, a cooling liquid return system 440is provided below the condensers to catch the falling liquid condensate462 in order to prevent the falling condensate 462 from coming intocontact with the HDDs 125 located below the condenser 460. In at leastone embodiment, as illustrated by FIG. 5, the condensate 462 ischanneled via a secondary conduit 545.

In one embodiment, the condensed cooling liquid collection system 440collects the condensed liquid as the liquid drops from the condensationsurface 465, due to gravity, following condensation of the rising vapor422. According to the illustrated embodiment, the condensation surface465 is angled to at least one side relative to a horizontal plane tocause the condensate 462 generated from the condensation of the risingvapor 422 to flow off towards at least condensed cooling liquidcollection system 440.

The condensation fluid (not shown) flows within the tubes/pipes close tothe condensation surface 465 and maintains the condensation surface 465at a lower temperature than a condensation point of the rising vapor422. According to one or more embodiments, the condenser 460 and inparticular the condensation surface 465 comprises at least one length oftubular piping extending from an external connection point running tothe inside of the enclosure 405 and then back to the outside of theenclosure 405. The external surface of the tubular piping can, in oneembodiment, provide the condensation surface 465, and the condensationfluid flows through the interior bore of the tubular pipe from anexternal fluid source/reservoir.

Indicated below liquid surface 420 in the lower volume 402 of enclosure405 is a power distribution unit (PDU) 425 sitting atop a platform 430.This platform 430 can be simply a ledge or space at which PDU 425 can benested. Alternatively, the platform 430 can be one specifically designedwithin immersion cooling tank 400 or as a part of server rack 410/510.PDU 425 is coupled via a power cable 470 to the external power source475 located on the outside of immersion cooling tank 400/500. PDU 425provides electrical power to server 200/300 as well as other electronicdevices within immersion cooling tank 400/500, of which HDDs 125 isillustrated. HDDs 125 is shown located above liquid surface 420 on aplatform 445, which can be porous (air accessible) in one embodiment. Afirst power connector 427 is shown extending from PDU 425 to server200/300, while a second power connector 528 extends to HDDs 125. In FIG.5, communication and data cable 305 is also shown connecting HDD 125 toserver 300 to enable data transfer between server 300 and the HDD 125during operation of both the server 300 and HDD 125 within immersioncooling tank 500.

Within immersion cooling tank 400/500, immersion server 200/300 isconnected via a number of connectors 415 indicated at the bottom ofserver rack 410/510. These connectors 415 allow servers 200/300 to beinsertiably coupled to the server rack 410/510. The connectors 415 arecoupled to or are extensions of a network cable bundle 450 which enablesimmersion server 200/300 to communicate with other devices both withinand outside of immersion cooling tank 400/500. In alternate embodiments,the communication and data cables of network cable bundle 450 can beconnected within the enclosure 405 by running the cable bundle 450through a trunking or sealed space (not expressly shown) that isprovided within the enclosure 405, such as within the enclosure walls.The cable bundle 450 is then connected to the backs of the server(s)200/300 via specific rack connectors 415.

For connections to endpoints outside of the enclosure 405, immersioncooling tank 400/500 includes network cable bundle 450 and power cable470, which extend through a side wall of enclosure 405 into the exteriorspace outside the tank 400/500. To support the interconnection ofmultiple server racks, for example, a large number of network cables arerequired to access the immersion cooling tank 400/500. These cablesrequire an opening to access the enclosure 405, and the size and numberof openings can significantly increase the opportunity for loss ofcooling fluid to the outside of the tank 400/500. Thus, communication inand out of the immersion tank 400/500 requires a design that enables thecommunication and data cables of network cable bundle 450 to enter theimmersion cooling tank 400/500 without providing any seams through whichthe immersion fluid can escape. Similarly, the design must allow for thepower cable 470 to be run into the tank 400/500 from the outside inorder to provide power to the devices inside of the tank 400/500,without allowing for escape of the cooling fluid. This aspect of thedisclosure thus provides a robust solution to seal the openings createdby the numerous cables and any other openings that may be requiredwithin the side walls or cover of the immersion cooling tank 400/500.

In order to support the extension of network cable bundle 450 and powercable 470 from the outside of immersion cooling tank 400/500 to theinside, and vice versa, one aspect of the disclosure provides arubber-based grommet 455 that operates to seal the areas at which therespective cables enter and/or exit the wall of the cooling tank400/500. The rubber grommet 455 fits tightly around Cat 6 or fibercables and power cables, and the rubber grommet 455 is then insertedinto an opening in the wall of the immersion cooling tank 400/500. Onceinserted into the opening, the rubber grommet 455 maintains a tight sealaround the cables and the perimeter of the opening and prevents escapeof vapor from inside the tank to the outside air and vice versa. Thiscapability of preventing escape of the cooling fluid is an importantaspect of the immersion tank 400/500 design.

FIG. 6 presents various different views of the configuration of thegrommets 455. FIG. 6 also illustrates the use of the grommets 455 withmultiple cables, representing network cable bundle 450, extendingthrough the wall of the immersion cooling tank 400/500. An exterior andan interior view of the two interlocking sides of a single grommet 455are shown by FIGS. 6A-6B. According to one aspect of the disclosure, andas illustrated by FIGS. 6C-6E, immersion cooling tank 400/500 isconfigured with one or more grommets 455 placed in both the internal andexternal surface of one or more of the walls 605 of the enclosure 405(FIG. 4) to create a seal through which communication and data cables ofnetwork cable bundle 450 and power cables 470 can be introduced into theenclosure 405 from the outside. FIG. 6C shows the stacking of aplurality of grommets 455 inserted into the wall 605 of enclosure 405and extending out of the interior wall surface 615. FIG. 6D thenillustrates network cable bundle 450 extending through the grommet 455at the exterior wall surface 610 of the enclosure 405, while FIG. 6Eillustrates network cable bundle 450 extending through the grommet 455at the interior wall surface 615 of the enclosure 405. According to theillustrated embodiment, the network cable bundle 450 represents 64infiniband connectors. However, it is appreciated that the use ofgrommets 455 can be applied with other types of network and powercabling. Importantly, the configuration of the grommet 455 and thematerial utilized to construct the grommet 455 allows the various cablesto be introduced into the enclosure 605 without causing leakage ofeither the cooling liquid 412 or the vapor 422 to the outside of theenclosure 605. These special rubber-based grommets 455 are utilized toenclose the point(s) of penetration into the tank from the outside toprevent (1) leakage of the dielectric fluid and/or (2) leakage of thehigh pressure vapor above the liquid surface of the enclosure.

Thus, grommet 455 has specific qualities that enable the cables'accesses to be sealed to prevent both liquid and/or vapor from escapingto the outside of enclosure 405. Generally, the selection of the grommetmaterial requires consideration of (1) the qualities of the materialthat makes it flexible enough for use but not prone to allow for leakageof liquid or vapor, even under high pressure, (2) durability of thematerial under constant heat conditions, (3) the specific cooling fluidbeing utilized within the immersion tank, not having any negativeinteractive properties with the grommet material, and (4) other criteriathat can be relevant to the system designer. According to one or moreembodiments, butyl rubber exhibits each of the above characteristics andis thus provided as the immersion grommet 455 in example immersioncooling tanks 400/500 as well as the other immersion cooling vesselsdescribed hereinafter.

In one or more embodiments, the dielectric liquid 412 utilized as thecooling liquid is Novec fluid, which has limited reactive propertieswith butyl rubber. Thus, the utilization of the butyl rubber grommet 455provides a solution that maintains flexibility while in contact with theNovec vapor. The use of the butyl rubber provides a rubber grommet 455that fits tightly around the cables, e.g., Cat6 or fiber cable (see600D-600E). And, the rubber grommet 455 maintains a tight seal betweenthe vapor inside the enclosure 405 and the outside of the enclosure 405.This aspect of the disclosure thus provides a robust solution to sealthe openings required to run the numerous network and other cables.Additionally, utilization of the grommet 455 enables and/or facilitatesfuture upgrades and cable replacement throughout the working life of theenclosure 405, without having to glue or epoxy the cables into the tank.

According to one aspect of the disclosure, immersion cooling tank400/500 includes a dielectric cooling liquid 412 that is selected basedon the fluid exhibiting certain desirable characteristics with respectto its high volatility (low boiling point temperature) when exposed tosurface heat dissipating from an operating server 200 and its lowcondensation barrier from vapor to liquid when exposed to roomtemperature condensation liquid, such as water. In one or more of thedescribed embodiments, the dielectric cooling fluid is Novec 649, aproduct of 3M®. Novec 649 is a dielectric fluid that boils at 49 degreesCelsius. More importantly, the Novec fluid does not conduct electricityand/or does not react or interact with surrounding components whenexposed to electricity or electrical components. Another benefit to theuse of Novec fluid is its superior functioning during servicing ofcomponents. During servicing of a server, for example, once the serveris pulled out of the fluid and left to sit for a few seconds, the highvolatility of the fluid causes the Novec 649 fluid to evaporate leavinga dry surface of the server to work on.

As one aspect of the disclosure, and with the use of Novec liquid as thecooling liquid, the condensation fluid can be surface water at roomtemperature. As introduced above, Novec liquid is highly volatile andhas a boiling point temperature of 49 degrees Celcius. Thus, fromexperimentation it has been shown that a single server operating minimalprocesses dissipates sufficient heat to raise the temperature of theNovec liquid to its boiling point temperature, resulting in vaporizationof a portion of the liquid. With such a low boiling point, the thresholdfor cooling the vapor to its condensation point is relatively low. Thus,within one or more embodiments, water provide at room temperature can beutilized as the condensation fluid. Because the boiling point of wateris not achieved until the water is heated to 100 degrees Celsius, water,when utilized as the condensation liquid, provides a vast amount of heatabsorption capacity to condense the rising Novec vapor. Additionally,water has a high specific heat and relatively high thermal diffusivitywhen compared to other liquids, making water an ideal candidate for useas the condensation liquid. Further, there is an abundant supply ofwater and water is relatively inexpensive as a condensation liquid.

According to one aspect of the disclosure, the dielectric vapor risesand the condensed cooling liquid falls due to relative density andoperation of gravity. Thus, the cooling of the electrical components canhave system or server power usage effectiveness (SPUE) of 1.00. SPUErefers to the ratio of total amount of power used by the data centerfacility to the power delivered to the IT equipment.

As a next benefit, the implementation of the immersion cooling tankprovides an opportunity to increase the density of IT gear in a serverrack (tank). When servers 200/300 are operated within immersion coolingtank 400/500, there is no longer the need for large, space consumingheat sinks, as the dielectric fluid boils directly off the processorchip. Thus, the server and/or rack space or volume that was once usedfor fans or heat sinks for the server module can now be filled withuseful functional components, such as other processors.

As yet another benefit, the dielectric fluid provides large thermalcapability, as the thermal capability of the dielectric fluid issignificant. Thus, the components in the dielectric fluid can operatewell below the thermal limits. Thus, a conventional server system whichwould only support low wattage processors due to the limited volumeavailable for heat sinks can now support high end processors.

Finally, with the implementation of the immersion cooling tank 400/500,there is no longer a need for expensive data center cooling equipment.The immersion tank is a sealed unit and because all dissipated heat iseventually absorbed and transmitted away from the enclosure by thecondensation fluid, there is also no need to cool the room around thetank. Also, the immersion tank can operate in almost any environment anddoes not require an air-conditioned or heated space.

C. Vapor Cooling of HDDs within Immersion Cooling Tank

One consideration that is relevant to the described innovation includesan appreciation of the limitations of operating HDDs within a liquidenvironment, such as an immersion-based cooling system for datacenterservers, as presented herein. Conventional HDDs are designed with alubricated rotating spindle which needs to be free from obstructionwhile spinning in order to operate effectively and enable a longerlifespan of the HDD. As determined by practical observation and testing,engineering analyses, and/or extrapolation of known theories in physicsand other sciences, submerging a HDD in a liquid filled enclosure doesnot properly accommodate cooling needs of rotating HDDs by virtue ofparasitic friction that is induced when liquid enters the HDDmotor/spindle region. This entering liquid (a) dissolves lubricant usedon the spindle over time and (b) increases the amount of friction on themoving parts, thus causing increased wear and tear, which results inshorter lifecycles of the HDDs. Additionally, the above effects alsoincrease the need for frequent maintenance of the HDDs, which negativelyaffects the equilibrium achieved and/or desired within an operatingimmersion cooling system.

Notably, to address this limitation with HDDs utilized withimmersion-based datacenters, a conventional solution would includeutilizing remote storage racks that would be in a different physicallocation. These remote storage racks then require a fan-based coolinginfrastructure, typically involving additional space, and use of largenumber of fans powered by an external power source. This solution alsorequires large amounts of cabling from the data center to the remotestorage location, which along with the increased power consumption,significantly increases the expense of running and/or cooling the datacenter and storage facility. Another conventional solution involvesexpensive HDD encapsulation with epoxy to prevent liquid from enteringthe internal structure of the HDD. Among the problems and/ordisadvantages with this approach are: (1) the approach is relativelyexpensive, as the epoxy is costly and the process requires additionalamounts of man/machine hours; (2) the epoxy encapsulation is permanent,thus preventing any future maintenance on the device; (3) epoxyencapsulation voids the HDD manufacturer's warranty; and (4) theencapsulation in epoxy increases the dimensions of the resultingencapsulated HDD, which makes the encapsulated HDD difficult to fit intothe standard HDD carriers (such as drive space 235 (See FIG. 2)).

One aspect of the described embodiments thus presents a method andsystem to effectively deploy an immersion based solution for a serverthat utilizes rotating HDDs, while accommodating the cooling needs ofrotating HDDs, without the additional expenses and other limitationsand/or problems inherent with the above conventional approaches. It isalso appreciated that the implementation of this configuration ofservers enables development and deployment of ultra-dense liquid cooledservers as the HDDs do not have to be accommodated on the serverchassis.

The example immersion cooling tanks 400/500 of FIGS. 4 and 5 representstwo-phase heat transfer HDD cooling system that operates as avapor-based cooling system for HDDs, according to one or moreembodiments. As provided by FIG. 5, the interior of enclosure 405 alsoincludes a HDD cooling area 525 in which at least one HDD 125 can beplaced during operation of the HDDs 125. The HDD cooling area 525 islocated at a first distance above the liquid surface 420 withinenclosure 405 and is in a direct path of plumes of vapor 422 rising offthe liquid surface 420. Thus, the above introduced immersion coolingtank 400/500 provides techniques for cooling hard disk drives (HDDs)within an immersion cooling environment without having to directlyimmerse the HDDs within the cooling liquid medium. This techniqueinvolves placing the HDDs in the path of a rapid flow of rising vaporgenerated by boiling of a highly volatile fluid in a cyclical multiphase(i.e., two-phase vaporization-condensation) heat transfer enclosure orvessel, such as an immersion cooling tank 400/500. The two phase(vaporization-condensation) tank 400/500 is configured with a liquidimmersion system that boils the cooling liquid 412 within a bottomenclosure or reservoir and creates vapor plumes 422. The vapor plumes422 are driven by density gradients through a HDD cooling area 525located above the cooling liquid 412 within the tank 400/500. As thevapor 422 rises, the vapor 422 passes with an upward velocity over theexposed surface of the HDDs 125 located within the HDD cooling area425/525 and the vapor 422 flows across the exposed heated surface areaof the HDDs 125. The movement of the cooler vapor 422 across and awayfrom the surface of the HDDs 125 causes the rising vapor 422 to absorbsome of the heat being dissipated from the surface of the HDDs 125 aswell as ambient heat generated by the HDDs 125 within the HDD coolingarea 425/525. Thus, the vapor 422 cools the HDDs 125 while in transit toan upper condenser 460.

At the upper condenser 460, the vapor 422 is condensed back to liquidcondensate 462 by coming into contact with the condensation surface orcondensation pipes/tubes in which the condensation liquid flows. Thecondensation of the rising vapor 422 occurs as the cooler condensationfluid flowing within the condenser 460 absorbs the heat energy from thevapor 422, causing the vapor 422 to convert (i.e., undergo a phasechange) from gas to liquid phase. The resulting cooling liquidcondensate 462 is then channeled via a secondary conduit 545 back to thecooling liquid reservoir in the bottom of the enclosure 405 to avoid theliquid condensate 462 coming into contact with the HDDs 125. Accordingto one or more embodiments, as illustrated by FIG. 5, the HDD coolingarea 525 includes a rigid, mesh type structure extended across onesegment of the inner perimeter of the enclosure above the liquid surface420 to form a porous platform 542 on which the one or more HDDs 125 areplaced. The platform is porous to allow the rising vapor 422 to passthrough the HDD cooling area 525 up towards the upper condenser 460. Inat least one embodiment, the HDDs 125 can be spaced apart from eachother and can be oriented within the HDD cooling area 525 to maximizethe amount of exposed surface that will come into contact with the flowof rising vapor 422. In a second embodiment with locally external HDDs,the HDD cooling area 525 includes a series of holding clips and/orsleeves designed to hold or support one or more of the HDDs 125. TheHDDs 125 are then suspended (i.e., held in place) within the HDD coolingarea 525 based on the location and/or configuration of the specificmechanism being utilized to hold the HDDs 125. It is appreciated thatother embodiments are possible in addition to those described herein,without limitation. According to one or more of the describedembodiments, the liquid that is utilized to produce the cooling viavaporization and condensation within the enclosure is a dielectricfluid. Selection of a dielectric fluid allows for the avoidance of anyelectrical interaction of the components being cooled with the fluidand/or rising vapor, among other benefits.

The HDD cooling system includes a heat source or heat dissipatingcomponent that dissipates heat into the lower volume 402 of theenclosure 405. Within the presented examples, the heat dissipatingcomponent is represented as a server 200/300. However, it is appreciatedthat many other types of heat dissipating components that can benefitfrom liquid cooling by immersion within the liquid as well as a genericheat source utilized solely to heat the cooling liquid to a boilingpoint can be utilized in place of or in addition to server 200/300within the HDD cooling system. According to one aspect of thedisclosure, regardless of the type of heat source or heat dissipatingcomponents utilized/provided, the amount of heat dissipated issufficient to heat the cooling liquid 412 within the lower volume 402 toa boiling point temperature at which at least a portion of the coolingliquid 412 evaporates, generating a plume of rising vapor 422. The plumeof rising vapor 422 flows/moves rapidly upwards through the HDD coolingarea 525 and across one or more surfaces of the at least one HDD 125.The moving vapor 422 cools the at least one HDD 125 via convection asthe vapor 422 comes into contact with and moves across the one or moresurfaces of the at least one HDD 125.

As introduced above, the HDDs 125 can be communicatively coupled to oneor more processing components that are either internal to the enclosure405 or external to the enclosure 405. According to one aspect, the HDDs125 can be respectively connected to externally and/or internallysupported server(s) via cables of network cable bundle 450 or HDDconnecting cable 305 (FIG. 3). It is appreciated that these connectors(450/305) are rated to operate at temperatures equal to or exceeding amaximum temperature from among the higher of (a) an ambient exteriortemperature of the HDDs 125 within the HDD cooling area 525 and (b) thetemperature of the rising vapor 422 and/or the boiling point temperatureof the cooling liquid 412.

According to the described embodiments, server 200/300 represents theheat dissipating component that dissipates heat into the lower volume ofthe enclosure 405 while the server is operating. One aspect of thedisclosure provides that the amount of heat dissipated by server 200/300is sufficient to heat the cooling liquid 412 within the lower volume 402to a boiling point temperature at which at least a portion of thecooling liquid 412 evaporates, generating a mass or plume of risingvapor 422. The plume of rising vapor 422, indicated by the verticalarrows, flows/moves rapidly upwards through the upper volume towardscondenser 460. According to one embodiment, the upper volume includes aHDD cooling area 525 (generally including a platform or other holdingstructure) in which at least one HDD 125 can be placed during operationof the HDDs. The plume of rapidly rising vapor 422 moves through the HDDcooling area and across one or more surfaces of the at least one HDD125. The rapidly moving vapor 422 cools the HDD 125 via convection asthe vapor 422 comes into contact with and moves across one or moresurfaces of the at least one HDD 125.

According to one embodiment, the plume of vapor 422 are driven bydensity gradients through the HDD cooling area located above the coolingliquid 412 within the tank 400/500. As the vapor rises, the vapor passesthrough the HDDs located within the enclosure with an upward velocityand flows across the exposed hot surface area of the HDDs. It isappreciated that the presence of a HDD cooling area is an optionalenhancement that is not necessarily provided in different embodiments ofthe immersion cooling tank 400/500.

In addition to the described components which relate to the vaporizationand resulting cooling aspects of the cooling system, the upper volume404 of the enclosure 405 of cooling system also includes: a condenser460 above both the HDD cooling area 525 and the at least one HDD 125;and a cooling liquid collection and/or return system (440/545), aportion of which is located above the HDD cooling area 525. In one ormore embodiments, illustrated in greater detail in FIG. 5, the coolingliquid return system comprises condensed cooling liquid (condensate)collection system 440 located above the HDDs 125 and a condensate returnconduit (or channel) 545. In one embodiment, the condensate returnchannel 545 is provided close to the perimeter of the upper volume 404of the enclosure 405, although the exact placement can vary by design.The condensate collection system 545 is located above the at least oneHDD 125 so as to protect the HDDs 125 from having any of the condensedcooling liquid condensate 462 fall on the operating HDDs 125. Thecondensate collection system 440 collects the condensed liquid as theliquid drops from the condensation surface 565, due to gravity,following condensation of the rising vapor 422. The condensate returnchannel 545 provides a conduit which extends from the condensatecollection system 440 to below the HDD cooling area 525 into the lowervolume 402 of the enclosure 405, returning the condensate 462 to thelower volume of the enclosure 505, while avoiding contact between thecondensate 462 and the at least one HDD 125.

According to the illustrated embodiment, the condensation surface of thecondenser 460 is angled to at least one side relative to a horizontalplane to cause the condensate 462 that is generated from thecondensation of the rising vapor 422 to run off towards the perimeter ofthe enclosure 405 away from the HDDs 125 and towards condensatecollection system 440. The condensate collection system 440 is alsoangled to allow the collected condensate to run off towards condensatereturn conduit 545 that directs the collected condensate 462 towards thelower volume of cooling liquid 412, without allowing the condensate 462to come into contact with the at least one HDDs 125.

While the above described embodiment provides for the condensate returnsystem preventing the condensate from coming into contact with the HDDs,at least one alternate embodiment is provided in which the HDDs 125 canbe “drip” tolerant. Thus, for example, the HDDs may have an exteriorcasing that allows the HDDs to deflect dripping condensate withoutnegatively affecting the operation of the HDDs. Thus, with thesealternate embodiments, the HDD cooling system can be configured withouta bypass system for the condensate, and which allows some dripping ofthe condensate on the HDDs. Additionally, in one more embodiments, theHDDs can be hermetically sealed, which would allow the HDDs to be fullyor partially immersed while operating. It is appreciated, that even withsuch HDDs, the aspects of the disclosure allowing for the cooling ofthese immersible HDDs by rising vapor can still be valuable given thevery high cost of the cooling fluid (i.e., Novec fluid). Vapor coolingof the HDDs enables savings on the total cooling liquid volume requiredat installation. With these types of HDDs, as well, one implementationcan provide that the HDDs are only partially immersed so that the HDDsare cooled by a combination of liquid cooling and vapor momentumcooling.

FIG. 7 is a flow chart illustrating one example of a method by which animmersion tank system can be implemented and utilized to provide coolingfor HDDs by producing a high velocity vapor flow across a HDD coolingarea within a heat-dissipating system, such as an immersion cooling tank400/500, in accordance with one embodiment. Aspects of the flow chartcan be implemented with reference to one or more components describedwithin any one of the different embodiments of a vapor cooling systemfor HDDs as presented in FIGS. 4-5. Method 700 also includes one or morefeedback control operations that can be performed during the cooling ofthe HDDs 125. The method 700 begins at block 702 at which the coolingliquid is inserted into the lower volume 402 of the immersion coolingtank 400/500. The method 700 then includes submerging the processingdevices (e.g., server processor and memory modules) into the coolingliquid (block 704). At block 706, method 700 provides placing the atleast one HDD within a HDD cooling area 525 of the immersion coolingtank 500. As stated above, the immersion cooling tank is configured witha lower volume having a cooling liquid and a processing device submergedwithin the cooling liquid and which device dissipates heat sufficient toraise the temperature of the cooling liquid to a boiling pointtemperature and cause vaporization of a portion of the cooling liquid togenerate a plume of rising vapor 422. The immersion cooling tank 400 isfurther configured with an upper volume 404 in which the HDD coolingarea 525, at least one condenser 460, and a liquid collection and/orreturn system 440/545 are located. The HDD cooling area 525 is locatedat a first distance above a surface layer of the cooling liquid 412within the lower volume and in a direct path of the rising vapor 422.The HDD cooling area 525 provides an area at which the at least one HDD125 can be cooled during functional operation of the at least one HDD125. The at least one condenser 460 is located at a second distance thatis above both the HDD cooling area 525 and the at least one HDD 125. Thecondenser 460 includes a condensation fluid that flows proximate to thesurface of the condenser surface and which maintains the condenser 460at a lower temperature than a condensation point of the rising vapor422. A substantial portion of the rising vapor 422 that passes throughthe HDD cooling area 525 and cools the at least one HDD 125 is condensedback into liquid phase on contact with or exposure to the condenser 460.

The method 700 further comprises applying power to and/oractivating/initiating the operation of the submerged processing deviceto cause the processing device to execute one or more programinstructions (block 708). The processing device's execution of programinstructions causes the processing device to generate heat sufficient tocause the cooling liquid to boil and vaporization of the cooling liquidto occur. The vaporization of the cooling liquid results in generationof a rising vapor, sufficient to cool the HDDs.

Method 700 then includes monitoring, via one or more electronic sensorsand/or feedback devices, the operating conditions of the immersioncooling tank 400/500 (block 710). Without limitation on the disclosureand according to one or more embodiments, among the feedback and/orcontrol devices and systems that can be provided within exampleimmersion cooling tanks 400, 500 are: pressure sensors 540 andassociated feedback control system, a fluid level sensor 535 andassociated feedback control system, and a condensation fluid leakagedetection system, which includes conductivity strip 530 (FIG. 5). FIG.18 (which is further described in later sections) generally shows asecond embodiment of a rack-based immersion cooling tank 1800 configuredwith a plurality of additional sensors and feedback and/or controldevices. With the exception of the pressure regulating system describedwithin Section H, the mechanisms and methodologies utilized to providethe various system controls are only generally provided for herein.However, it is appreciated that these aspects of the disclosure involvetechniques related to general autonomic real-time control mechanismsand/or methodologies that provide for the proper operation andmaintenance of the entire immersion cooling system. These controlmechanisms/methodologies include several detection mechanisms/devicesand functional control loops to enable features such as, but not limitedto, controlling the immersion fluid level, detecting leaks in thecondenser, detecting and controlling to differential pressure and otherpressure conditions, detecting or responding to other conditions withinthe tank, and providing automatic power shutoff and/or transmittingnotification to administrator or IT personnel, in response to certaindetected conditions. One or more of the responses to a detectedcondition can be implemented by processor execution of code associatedwith one or more feedback control module(s) 170 (FIG. 1).

Returning to the flow chart, at decision block 712, method 700determines whether there are any conditions detected that require systemmaintenance or a control response. When no such condition exists, method700 loops back to block 710 where the sensors continue to monitor forthe occurrence of one or more conditions. However, in response to therebeing a condition detected requiring system maintenance or a controlresponse, method 700 includes mechanically, programmatically, orelectronically performing one or more corresponding control operationsto maintain an equilibrium or proper operating state of the immersioncooling tank 400/500 (block 714). With respect to the vapor cooling ofthe HDDs 125, operating conditions of interest include, but are notlimited to: (1) the amount of cooling being provided by the risingvapor, which can be controlled by throttling or increasing the amount ofprocessing being performed by the processing devices, which correlatesto the amount of rising vapor generated; (2) the amount of heat beingdissipated by the HDDs relative to the cooling being provided by therising vapor, where a rate of data access to the HDD can be controlled(e.g., throttled) if the HDDs are not being sufficiently cooled by therising vapor; and (3) the highest level of the cooling liquid within thelower volume relative to the HDD cooling area, where the cooling liquidis required to remain below the HDD cooling area to avoid contact withthe HDDs.

It is appreciated that operation of at least one processing device orserver results in heat dissipation into the surrounding cooling liquid,which absorbs sufficient heat to cause the cooling liquid to reach theboiling point and result in vaporization of some of the cooling liquidto create rising vapor. According to one embodiment, method 700 furtherincludes monitoring via a set of electronic and mechanical sensors andfeedback devices one or more operating conditions (e.g., as pressure,temperature, liquid level, etc.) within the immersion tank (block 710).

D. Method to Protect PDUs from Water Infiltration and Enhance PDUEfficiency by Immersion in a Dielectric Liquid

With the above described immersion cooling tank 400/500 operating as acyclical heat exchange ecosystem, which is sealed to prevent loss ofcooling fluid to the outside, one important consideration is the need toprovide power to the plurality of servers 200/300 and other electronicdevices (e.g., HDDs 125) operating within the immersion cooling tank400/500.

According to one or more of the described embodiments, the coolingliquid that is utilized to produce the cooling via vaporization andlater condensation within the enclosure is a dielectric fluid. Selectionof a dielectric fluid allows for the avoidance of any electricalinteraction of the components being cooled with the fluid and/or risingvapor, among other benefits. For cost and other considerations, watercan be utilized as the condensation fluid. Given this use of anon-dielectric condensation fluid, such as water, additionalconsiderations also have to be given to the fact that water isconductive and that there is a risk of leaks of the condensation fluidwithin the tank. Such a leak could cause water to come into contact withan exposed power distribution unit (PDU), which can cause electricalarcing and other potentially damaging and/or dangerous conditions withinthe tank. Because water is conductive, the proximity of the water tohigh voltage equipment creates UL (electrical standards board) safetyconcerns. Also, with water being utilized as the condensation liquid,the submerged PDU is protected from the water spillage in the event of aleak in one of the overhead condenser units. This is because the water,which is less dense than the dielectric cooling fluid, will simply floaton top of the dielectric cooling fluid, without coming into contact withthe PDU.

Thus, according to one aspect of the disclosure, rather thande-localizing either the condensation process or the power distributionunits from being local to the immersion cooling tank 400/500 or to eachother, or require that the condensation liquid be a non-conductivefluid, which is significantly more expensive than the use of water andprovides less heat absorption capacity, aspects of the presentdisclosure provides a solution that yields additional benefits to theimplementation of example immersion cooling tank 400/500. To furtherdescribe this implementation, reference is made to FIGS. 4 and 5 andlater to FIG. 8. This aspect of the disclosure provides techniques toprevent electrical arcing and/or other problems otherwise inherent if anon-dielectric condensation fluid falls on the power distribution unitswithin the immersion tank. Additionally, this aspect yields severalbenefits associated with power efficiency, as detailed herein.

Referring to FIGS. 4 and 5, in order to provide the required electricalpower, the immersion cooling tank 400/500 also includes powerdistribution units (PDUs) 425. As shown by FIGS. 4 and 5, PDUs 425 aresubmerged below the surface 420 of the cooling liquid 412. Providingpower to the submerged PDUs 425 are power cables 470, which extendthrough the walls of immersion cooling tank 400/500 for connection toexample external power source 475. PDUs 425 can provide electrical powerto the various electronic devices and/or components within immersioncooling tank 400/500 via power connectors (e.g., 427). As describedherein, servers 200/300 represent electronic devices and/or componentsthat are submerged below the surface 420 of the dielectric coolingliquid 412, and HDDs 125 represent electronic components and/or devicesthat are located within the tank volume, above the surface 420 of thedielectric cooling liquid 412.

Immersion tank 400/500 also comprises at least one condenser 460 (andpotentially multiple condenser sub-units, as illustrated and describedbelow with respect to FIG. 8) located above the surface 420 of thedielectric cooling liquid 412. Condensation fluid (not shown) flows inliquid form through the condenser 460 during normal operation of theimmersion cooling tank 400/500. Within the various describedembodiments, the condensation fluid (or liquid) is less dense (i.e., hasa lower density) than the selected dielectric cooling fluid. Leakage ofthe condensation fluid into the lower tank volume 402 results in thelighter condensation fluid floating atop the surface 420 of the heavierdielectric cooling liquid 412. The leaked condensation fluid would thusnot come into contact with the PDUs 425, as the PDUs 425 are submergedbelow the surface 420 of the cooling liquid 412. Thus, the potential forleaks that may occur within the upper condenser 460 leading to shortcircuits or other electrical problems within the immersion cooling tank400/500 is minimized and/or substantially eliminated.

According to the described embodiments, the PDUs 425 can provide eitherAC or DC power, depending on the requirements of the electronic devicesand/or components being powered. In one or more embodiments, the PDUs425 can also be blind-mated for additional service benefits. Also in oneembodiment, the immersion cooling tank 400/500 can be configured toinclude a power distribution system or Bus Bar type infrastructure thatis embedded into the server rack in order to enable hot pluggable powerto a server chassis that is subsequently inserted into the server rack.The power distribution system is generally presented by power connector415 with power cable 427 extending from PDUs 425; however, alternatemethodologies are possible for coupling power to the submergedelectronic devices requiring electrical power, including the utilizationof at least one of a PDU, transformer, inverter, and a power deliveryappliance, which can be either internal or external to the immersioncooling tank 400/500.

According to one aspect, submerging the PDUs 425 in the dielectricliquid provides an enhanced heat transfer coefficient to the powerconductors of the PDUs 425 due to the contact with the cooling liquid.This cooling, which minimizes temperature-induced resistances within the(power transporting) metal conductors of the PDUs, increases acurrent-carrying capacity of the power transport conductors and the PDUsand further results in a reduction in the internal power losses of thePDUs 425, greater power transfer efficiencies, and greater PDU capacity.

Each of the presented figures of immersion cooling tanks (e.g.,previously presented FIGS. 4 and 5 and later presented figures)illustrates the placement of PDUs 425 within example immersion coolingtank 400/500. In the illustration of FIGS. 4 and 5, PDUs 425 are placedin a side location of the immersion cooling tank 400/500 away from theactual rack space, and the PDUs 425 are then connected via power cables427 to the power connectors (generally shown) at the base of theserver(s) 200/300. Importantly, PDUs 425 are located below liquidsurface 420, and the determination of liquid surface level 420 andresponse mechanisms in place to monitor and/or control the liquid levelwould take the location of the PDUs into account to ensure the PDUs 425remain below the liquid surface level 420.

As illustrated by FIG. 18, power cables 1875 extend beneath the coolingliquid surface from the PDUs 425 to the powered devices, of which firsttwo servers 200 are shown connected. As further illustrated by FIG. 18,the feedback and/or control mechanisms within immersion cooling tank1800 can also include vertical thermistor array 1815, float or fluidlevel sensor 1820, which detects a current fluid level 1625 of coolingliquid within the immersion cooling tank 1800, conductivity strip 1825,and flow control valve 497 and external pipe connector 495 whichconnects the immersion cooling tank 1800 to a multi-rack immersionliquid distribution system, in one embodiment. One or more of thesecomponents and/or the collection of these components can be utilized toensure that the liquid level within the tank 400/500 remains within theacceptable range of cooling liquid volume required.

E. Partitioned, Rotating Condenser Units to Enable Servicing ofSubmerged IT Equipment Positioned Beneath a Vapor Condenser

As presented by the above descriptions, one aspect of the disclosureinvolves the recognition that a passive 2-phase immersion cooling systemrequires a condensing unit to be placed gravitationally above the heatdissipating servers. Another aspect of the disclosure provides a designof the immersion cooling tank that includes a lid structure comprised ofmultiple rotating condensers. With these designs, the condensers of theimmersion tank are configured as separate adjacent condenser basedcovers over separate vertical spaces. The separate condensers arereferenced herein as condenser sub-units, to indicate that eachrepresents a sub-part of a larger condenser or condenser system. Eachcondenser sub-unit is rotatable to an open position to expose and/orprovide access to a server blade or other electronic device locatedbelow that partition. The individual partitions allow forservicing/maintenance to be performed on an exposed server blade, whileallowing the system to continue operating and cooling the remainingimmersed server blades (including those directly adjacent to the exposedblade) using the remaining condensers that remain in place over therising vapor.

One existing solution to cooling server racks utilizes a condenser thatcoils around the outer rim of an IT rack for immersion cooled ITservers. This solution has inherent limitations with respect to thesolution's cooling capacity and the solution's ergonomic flexibility, asthe height of the cooling structure (rack or tank) must be grown toaccommodate the condenser coils. Also, existing solutions to thechallenge of accessing individual server nodes operating in an immersionenvironment are limited to routing the condenser tubes around the outerrim of the IT rack. With this design, access to the server nodes allowsfor a significant amount of vapor escape during service events andnecessitates that the rack footprint grow outwards—increasing overallcost and service time. Accordingly, another aspect of the disclosure isthe recognition that the conventional methodology parasitically limitsthe available condenser surface area for vapor to liquid phase changeduring even a minimal service, and thus limits the total coolingcapacity of the system. Once the cover to the tank is opened, the entirevolume of rising vapor escapes the system, resulting in loss ofvapor/liquid mass, which unnecessarily leads to significant coolingfluid replacement costs. The disclosure presents a more desirable andfunctional design that allows one to gain access to individual servernodes in a blade or multiple chassis rack without disrupting thecondensation process and/or while allowing only a minimal amount ofvapor to escapes during the service event.

FIG. 8 shows an example three dimensional rack-configured immersioncooling tank 800 designed to support multiple side-by-side electronicdevices, such as servers 200/300, within a server rack (not specificallyidentified), where at least a portion of the electronic devices aresubmerged in cooling liquid for cooling of the electronic componentslocated below the surface of the cooling liquid. Immersion cooling tank800 is a three dimensional view of a practical example of the schematicsof immersion cooling tank 400/500 and includes a tank volume (see FIGS.4-5) containing a dielectric fluid (not shown). For continuity,immersion cooling tank 800 shall be described with reference to featurespresented in FIGS. 4 and 5, where applicable. In FIG. 8, immersioncooling tank 800 is shown without its front panel, which is madetransparent (i.e., not visible) in order to provide a clear view of therack space, servers, and other components and features inside the tankvolume.

Immersion cooling tank 800 includes exterior casing 805 and a base panel810. The specific design of the casing 805 can vary from one embodimentto the next, and is thus not limiting on the disclosure. Included inbase panel 810 are one or more wheels 812 to enable the immersioncooling tank 800 to be moved along the ground surface on which theimmersion cooling tank 800 is located.

Additionally, immersion tank 800 includes a dry tank 820 coupled to anexterior side panel. Dry tank 820 can be utilized to allow otherair-cooled IT equipment to be physically coupled or co-located with animmersion-cooled IT rack. Dry tank 820 accommodates those other ITcomponents that are typically paired with volume servers within theresulting datacenter, but which components cannot be easily immersed inliquid. Examples of these liquid-incompatible devices or componentsinclude, but are not necessarily limited to, rotating HDDs and networkswitches. Another aspect of the design can include an overhead storagecompartment for HDDs, power, switches, etc. Thus, in one embodiment,rather than rule out immersion cooling for IT servers that require orinclude these types of components that are required to be locallycoupled, one aspect of the design of the immersion cooling tank involvesproviding a detachable, containment structure for “top of rack switches”to be paired with the immersion tank. These compartments can be placedon either side of the immersion rack, designed with attachments oneither or both sides, or overhead to save floor space. Also, thesecompartments can provide additional storage for large JBOD (just a bunchof disks/drives) or HDD clusters that cannot be immersed in liquid.

It is appreciated that one rack implementation that does not involve useof liquid-incompatible devices or components is a rack design for and/orrequirement that servers use SSD drives, “diskless” configurations, orexternal, remote storage solutions. The present design appreciates andresolves the problem of latency deficiencies inherent with use of remotestorage by ensuring the storage components are attached to the tank in alocally external configuration of the server. Also, the present designalso appreciates and resolves the problem of latency and increasedcabling costs when switches are mounted in centralized network switchtrees.

Immersion cooling tank 800 includes a plurality of electronic devicesand/or components, including servers 200/300 and other devices 830, someof which are rack mountable. As illustrated, a plurality of electronicdevices, including servers, can be aligned and/or inserted in aside-by-side configuration within the rack (not shown). FIGS. 4 and 5illustrate the example vertical orientation of these servers 200/300relative to the tank volume. With the exception of HDDs, where provided,these electronic devices can be submerged or partially submerged in thedielectric fluid.

In one embodiment, the collection of servers 200/300 represents a datacenter. Further, in this embodiment of immersion cooling tank 800, thecooling liquid (e.g., 412, FIG. 4) is a dielectric that has highvolatility (i.e., low boiling point) relative to the temperatureincrease of the cooling liquid caused by the heat dissipation from theoperating servers 200/300. Accordingly, during normal server operations,the operating servers 200/300 dissipate heat sufficient to raise thetemperature of the dielectric liquid 412 to the boiling pointtemperature of the dielectric liquid 412, and this rise in the coolingliquid temperature caused by the heat dissipation from the serverscauses the vaporization of the dielectric liquid (412) generatingdielectric vapor (422) as described with reference to FIGS. 4 and 5.

As further illustrated, immersion cooling tank 800 includes one or morePDUs 425, which provide the electrical power required by the electronicdevices to operate. As shown, PDU(s) 425 are located below the surfacelayer 420 (FIG. 4) of dielectric fluid. The PDU(s) 425 are connected tothe electronic devices and to an external power source via powercable(s) 875.

Immersion cooling tank 800 includes a tank cover 880 that is connectedto the tank via a hinge mechanism and which allows the interior volumeof the tank to be sealed. When the cover is placed over the tank, thetank is sealed to be air-tight, such that no vapor can escape throughthe seal created. Creation of this air-tight seal can involve the use ofa rubber or other impervious material along the perimeter edges of thetank cover 880 and/or the top of the lower tank volume at which the tankcover interlocks. Located within the tank cover 880 is a bellows 890. Asdescribed in greater details in Section I, bellows 890 serves as a vaporpressure regulator during operation of immersion cooling tank as atwo-phase heat exchange cooling vessel.

As indicated by FIGS. 4 and 5, the various embodiments of the disclosurepresent a condenser 460 that is used to extract heat from the risingvapor from a liquid to vapor phase change system. As with theseillustrations, during operation of the immersion cooling tank 800, aportion of the dielectric cooling liquid within the lower volume of thetank evaporates to create a plume of rising vapor (422). According toone embodiment, the tank is configured such that a condenser (generallyrepresented as condenser 860 herein) is placed gravitationally above theheat dissipating servers, in the direct path of the rising vapor (422).The use of the cover assembly including the condenser provides asubstantially impervious seal for the tank volume to ensure that therising vapor within the tank volume cannot escape the tank volume whilethe condenser 860 is fixably rotated to a closed position over thespecific vertical space of the lower tank volume that is below thecondenser 860.

As further provided by the example embodiment of FIG. 8, the condenser860 is designed as multiple condenser sub-units 860A-H within the uppervolume 404 and located below the tank cover 880. Each condenser sub-unit860A-H represents a partition of the overall condenser 860 and isconnected to the tank casing via a rotatable hinge 880. The rotatablehinge 880 enables each condenser sub-unit 860A-H to be rotated a numberof degrees (e.g., 90-180 degrees) away from a fully closed position. Inone or more embodiments, the individual condenser sub-units 860A-H arerotatable from a closed position of 0 degrees to an open positionranging from greater than 0 degrees to a maximum number of degrees,where the maximum number of degrees is sufficiently large to allowaccess to the vertical space below the condenser sub-unit within thetank volume to access and/or remove a server or other device that isplaced within that vertical space of the tank volume.

According to one aspect, each individual condenser sub-unit 860A-Hincludes a separate condensation surface 865 from each other individualcondenser sub-unit and a separate local conduit 867 for providingcondensation fluid to cool the separate condensation surface. During anopening of a first condenser sub-unit (e.g., 860F), each otherindividual condenser sub-unit that remains in a closed positioncontinues to provide condensation of the rising vapor from the lowertank volume, while the first condenser sub-unit (860F) is open. Thus, anopening of the first condenser sub-unit (860F) does not hinder orprevent ongoing condensation from occurring at each adjacent secondcondenser sub-unit (860E, 860G) and the other non-adjacent condensersub-units (860B-860D, 860H) that remains in the closed position.

Thus, according to the above introduced aspect of the disclosure, theimmersion cooling tank 800 includes: a tank volume comprised of sidewalls and a base that allows a cooling fluid (412) to be maintained andheated therein; a rack structure within the tank volume having serverrails that supports removable insertion of a server-based informationhandling system (e.g., server 200/300); and a cover 880 that enclosesthe tank volume and which is designed with a condenser 860 configured asa plurality of condenser sub-units 860A-H. Each of the plurality ofcondenser sub-units 860A-H is rotatably connected via a hinge mechanism870 to the tank wall or several rails (or other fixed component) and canbe individually rotated from (1) a closed position in which a verticalspace below the condenser sub-unit 860A-H within the tank 800 is sealedto allow condensation to (2) an open position in which exposure and/oraccess to the inside of the vertical space is provided. The hingedmechanism 870 enables each individual condenser sub-unit (e.g., 860F) tobe opened independent of the other sub-units (e.g., 860G), and eachother condenser sub-unit can remain in a closed position while any oneof the condenser sub-units is opened.

This partitioning of the condenser 860 into condenser sub-units 860A-Henables access to one or more of (1) a particular HDD 125 physicallylocated below the particular rotatable condenser sub-unit 860A-H; or (2)a particular server 200/300 that is inserted into the server rack at avertical plane or location relative to the directional run of theservers within the immersed server rack. FIG. 8 illustrates thehorizontal run of side-by-side servers, each extending upwards intorespective vertical spaces. A separate condenser sub-unit 860A-H islocated above each separate vertical, such that the correspondingcondenser sub-unit 860A-H is located above a particular server 200(inclusive of HDDs) and/or server 300 and locally extern HDDs 125.

Illustrated at the back of immersion cooling tank 800 are a plurality ofnetwork cables 850, representing network cable bundle 450 of FIGS. 4 and5. As described above, one or more of network cables 850 are coupled tothe communication and data connectors of one or more servers 200/300 orother electronic devices 505 (and/or HDD 125, if provided). Also,illustrated at the back left section of immersion cooling tank 800 areintake and outflow valves 856A and 856B respectively connected to intakepipes 855A and outflow pipes 855B through which condensation liquidflows.

Additionally, four insets, labeled A-D, are presented in FIG. 8. InsetsA and B illustrate and/or provide additional details about possibleinternal configuration within vertical spaces below condenser sub-units,while insets B and C illustrate example make up of the condensersub-units themselves. Specifically, inset A illustrates example server200 placed within the server rack space extending vertically belowcondenser sub-unit 860A, while inset B illustrates example server 300with externally local HDDs 125 placed within the server rack spaceextending vertically below condenser sub-unit 860F of immersion coolingtank 800. As shown by the insets C and D of FIG. 8, described hereafter,each condenser sub-unit 860A-H includes a condensation surface 865 and acondensation fluid (not shown) that flows proximate to the condensationsurface 865 through pipes 867. The configuration of the pipes 867 withincondenser subunits 860A-H can be a simple loop as illustrated by pipes867A of inset C or can be much more intricate as provided by pipes 867Bof inset D. The latter configuration of pipes 867B with inset D providesa greater surface area that is exposed to the rising vapor (422) forcondensation thereof. The external surface of the tubular piping can, inone embodiment, provide the condensation surface for the condenser 860or condenser sub-units 860A-H.

In one embodiment, the condensation fluid flows through the interiorbore of the tubular pipe from an external fluid source/reservoir. Inanother embodiment, the heat exchange with the condensation fluid occurslocally via a radiator type structure attached to the exterior of theimmersion cooling tank 800 such that no external piping is required. Inyet another embodiment, as illustrated by FIG. 12, a stand aloneimmersion cooling tank can be configured with the exterior atmosphericair operating as the condensation fluid that cools the rising vapor(422). These two latter implementations enable aspects of the standalone immersion cooling tank described in the following section.

According to the above described aspects of the disclosure, the singularrack condensing unit position at the top of the immersion cooling tankis designed as and/or partitioned into a plurality of sub-condensersthat can be individually serviced or rotated to enable access to eachindividual server chassis (first node) without interrupting thecondensation process of the adjacent nodes. While described as asingular assignment of partitions to server nodes, or vice versa, thegranularity of the partitions is a design choice. As such, in otherembodiments, two or more server nodes can be located beneath a singlecondenser partition so long as the system has at least one additionalpartition and one addition server node located beneath that oneadditional partition. As designed, the individual partitions of thecondenser are each capable of being rotated at least a minimum number ofdegrees (e.g., 90 degrees) from a closed position to enable servicing ofIT equipment during runtime, without interrupting the flow of liquidthrough the condenser surface and/or requiring any other uptimeinterruption of the data server. As demonstrated by FIG. 8, with theimmersion cooling tank 800, while one server is exposed by opening thecondenser sub-unit (e.g., 860A or 860F) vertically above the server, theother servers remain submerged in a cooling fluid. The cooling fluidcontinues to boil and generate rising vapor, which then condenses on theclosed condenser sub-units 860B-860E, 860G-860H) to perpetuate the heattransfer cycle, while the exposed server(s) (e.g., below condensersub-units 860A, 860F) can be serviced.

In one or more embodiments, the number of individual condenser sub-unitsis numerically correlated to a fixed number of servers that can beaccessible via the opening of the individual condenser sub-unit. Thus, asingle server or a plurality of servers can be located within thevertical space below the condenser sub-unit. Also, a power switchingunit (PSU), of HDD, or other electronic device being cooled viavaporization-condensation fluid cycles can also be located within thevertical space.

F. Condensation Liquid Distribution System and Thermodynamic Stepping ofMultiple Working Fluids to Provide Cooling of Target Space

According to one or more embodiments, condenser sub-unit 860A-H (FIG. 8)and in particular the condensation surface 865 comprises at least oneextended length of tubular piping extending from an external piperunning to the inside of the enclosure via an intake path and then backto the outside of the enclosure via an out flow path. According to oneembodiment, the external ends of the tubular piping 855A-B (FIG. 8) ofimmersion cooling tank 800 can be connected with an externalcondensation fluid source, which can be working fluid reservoir 920(FIG. 9, described hereafter) to create a condensation loop, with thecondensation fluid being a first working fluid. In one or moreembodiment, immersion cooling tank 800 also includes a condensationliquid distribution system connected to and/or including the condensersub-units 860A-H. Condensation liquid distribution system includes anetwork of pipes 855 that run into and out of each condenser and/orcondenser sub-unit 860 and connect to a main piping system, which caninclude connecting end valves, for facility water connection. Two endvalves are illustrated, consisting first of intake valve 856A, at whichcondensation fluid is received from the external reservoir and passed tothe condenser sub-units 860A-H. The second valve is the outflow valve856B at which the heated or evaporated condensation fluid passes backtowards the condensation fluid reservoir on the outside. Notably, asillustrated by the inset figure showing an internal makeup of condensersub-unit, each sub-unit can consist of a single loop of piping 867proximate to a “separate” condensation surface 865. The actual number ofsuch loops of piping 867 can vary depending on the actual size of eachcondenser sub-unit, and the inset C and D are provided solely forexample to contrast with the larger condensers illustrated within FIGS.4 and 5.

According to at least one embodiment, the internal tubes or pipes of thepreviously introduced condensers 460 (FIGS. 4-5) and condenser 860 (FIG.8) and/or condenser sub-units 860A-H are coupled to the condensationfluid distribution system. The condensation fluid distribution systemcan in turn be connected to an external reservoir from which thecondensation fluid flowing into the condenser is provided. Thecondensation fluid is piped into the condenser 860 at a temperature thatis less than the cooling point or condensation temperature of the risingvapor 422. Within the condenser 860, the condensation fluid maintainsthe condensation surface 865 at a lower temperature than a condensationpoint/temperature of the rising vapor (422).

According to one or more embodiments, the condensing fluid utilizedwithin the condensers 460/860 can be water. With the known thermalproperties (i.e., vaporization point and condensation point) of thespecific dielectric fluid utilized (e.g., Novec fluid), regular watercan be utilized. Importantly as well, the water does not need to bechilled or cooled to provide the condensation of the rising vapor and assuch can be pulled from a facility water source reservoir. Thus, ambientwater is received from a facility water source (or reservoir) ascondensation liquid that condenses the rising vapor (422) from thedielectric fluid. The latent heat transfers from the vapor (422) to thewater, and the water is eventually returned back to the facilityreservoir. Notably, in one embodiment, this heated water can then beutilized for other uses, such as to heat the facility or otherstructure, etc., within an example environmental application.

FIG. 9 shows an example of one possible implementation of anenvironmental application that provides secondary heating based on theheat absorbed by the condensation fluid flowing through condensers460/860. Server 200 represents a heat source that is submerged in animmersion cooling tank (not shown) and operated while submerged below acooling liquid to cause vaporization of the cooling liquid. Examplecondenser 460 is a two phase condenser which receives condensationliquid at the intake pipes. Condenser 460 absorbs the heat from therising vapor (V1) and condenses the vapor back to liquid (L1), whichreturns to the immersion cooling tank. According to the illustratedembodiment, the amount of heat absorbed from the rising vapor (V2),i.e., the latent heat dissipation required for heat change from vapor toliquid condensate, can be sufficient to evaporate the condensationliquid (L2) into a condensation vapor (V2) as the heat is absorbed bythe condensation liquid flowing within the condenser. This condensationvapor can then be forwarded to a facility pump 910, which pushes thecondensation vapor (V2) towards a heat recovery system 920 for abuilding or other location requiring heat. Heat recovery system 920 ofthe building can include a third condensation liquid (L3) which canabsorb the heat from the condensation vapor (V2) and in turn vaporize togenerate a third condensation vapor (V3). With the latent heat removed,at least a portion of the condensation vapor can condense back to acondensation liquid (L2). This condensation liquid is then sent to anevaporative cooling tower 930, where any additional latent heat isremoved from the received fluid such that the remaining vapor condensesback to condensation liquid. The cooling tower 930 operates as thefacility water source reservoir in this example.

Turning now to FIG. 10, there is illustrated an example fluid-based heatexchange system that can be utilized with an immersion cooling tank400/500/800 to allow for stepped heat exchange via a plurality ofdifferent working fluids. FIG. 10 generally illustrates a method andsystem for providing cooling of a target space, i.e., the immersioncooling tank 1000, in which heat is being dissipated. Specifically, thefigure illustrates a method and system for cooling components disposedwithin an immersion tank that supports two-phase cooling of an immersionserver via an vaporization-condensation cycle utilizing a volatile(i.e., low boiling point) immersion liquid. In FIG. 10, a first dashedbox (first fluid transformation 1005) represents an example heatexchange components of an immersion cooling tank 700, where a coolingliquid 412 is evaporated into vapor 422, which vapor is then condensedinto a condensate 462 that returns to the cooling liquid volume. Asecond dashed box (cooling liquid distribution system 1010) representsthe fluid exchange or replenishment between a cooling liquid reservoir1020 and the volume of cooling liquid 412 within the immersion coolingtank 1000. The methodology for supplying cooling liquid 412 to theimmersion cooling tank 1000 can vary, and one example is provided withinSection H, described hereafter. A third dashed box (condensation liquiddistribution system 1015) then represents the heat exchange system,which includes a first heat exchanger or condenser 860 and can include aworking fluid reservoir 1030.

The heat exchange system diagram of FIG. 10 illustrates both sides of atwo-phase vaporization-condensation cooling system 1000, which includesa cooling liquid distribution system 1010 and a condensation liquiddistribution system 1015. Two-phase vaporization-condensation coolingsystem 1000 provides a first tandem of vaporization-condensation fluidtransformations 1005 within an example immersion tank, which is assumedto be immersion tank 800 (FIG. 8) in this example. With this first fluidtransformation 1005, a volume of cooling liquid 412 is evaporated intovapor 422 and the vapor is then condensed back into cooling liquidcondensate 462 by a condenser 860, which represents a first heatexchanger (Ex_1). Condenser 860 has a surface 865 at which the twoworking fluids are able to come into proximity with each other in orderto allow for the heat exchange from vapor 422 to condensation liquid1022 to generate condensation vapor 1024. The volume of cooling liquid412 can be obtained from a cooling liquid reservoir 1020, which servesto replenish the level of cooling liquid 412 within the immersion tank800 in the event of a leak and/or loss of fluid from immersion tank 800.According to one embodiment, a sensor and/or feedback mechanism thatincludes an intake flow control valve (e.g., 497 (FIG. 4)) attached tothe piping from cooling liquid reservoir 1020 can be provided with theimmersion tank 800 to automatically maintain the fluid levels in theimmersion tank 800 at a desired level.

In FIG. 8, a first heat exchanger is represented by condenser 860. Thecondensation which occurs within first heat exchanger involves a flow ofcondensation liquid, which occurs within condensation liquiddistribution system 1015. As shown by FIG. 10, condensation liquiddistribution system 1015 includes a condensation liquid reservoir 1020connected via a system of pipes, which are generally indicated as inflowpipes 1022 and outflow or return pipes 1024. It is appreciated that thesystem of pipes necessarily includes intake and return pipes 855A-B ofcondenser 860 (FIG. 8). Condensation fluid flows from condensation fluidreservoir 1030 and enters condenser 860 as a liquid having a lowertemperature relative to the vapor temperature and/or the ambienttemperature within upper volume of the immersion cooling tank 800. Inone embodiment, the condensation liquid absorbs heat form the vapor 422causing the vapor 422 to condense back into cooling liquid condensate462. The heat absorbed by the condensation fluid increases thetemperature of the condensation fluid and generates heated condensationfluid, which is returned to the condensation liquid reservoir 1020 forcooling, in one embodiment. In another embodiment, the condensationfluid enters the condenser in liquid form and the amount of heatabsorbed by the condensation liquid is sufficient to boil thecondensation liquid. This boiling results in a phase change fromcondensation liquid to condensation vapor, which can be returned to thecondensation liquid reservoir 1020 for cooling, in one embodiment.

According to one embodiment, condenser 860 can be referred to as heatexchanger1, to indicate alternate implementations in which multiplevaporization-condensation cycles are chained together between theimmersion cooling tank 800 and the “final” reservoir 1030 or final heatexchange medium. Thus in FIG. 10, the dots that extend left of thereservoir 1030 indicate that additional heat exchangers and associatedworking fluids can be included within the chain, prior to final coolingat the reservoir 1030 or other form of final heat exchange medium, suchas the atmosphere. Each subsequent heat exchanger would then utilize theheated condensation liquid of the previous heat exchanger as the heatsource that ultimately provides the required heat energy at that heatexchanger which the heat exchanger extracts and provides to heat theintake of condensation fluid by the heat exchanger. Implementation ofthis more complex heat exchange methodology can require use of differentcondensation fluids as the working fluid at each heat exchanger, wherethe different boiling and condensation temperatures allow for a phasedheating and cooling cycle until the final heat exchanger or reservoir1030. It is appreciated that the final heat exchanger can be theatmosphere, where the heated working fluid is cooled by contact withregular air.

According to another aspect, and as shown in greater detail by the insetdrawing of FIG. 10, which is further exemplified in FIG. 11, a pluralityof heat exchanges are arranged in tandem with each other. Each heatexchanger can have a different working fluid. Alternatively, two or moreof the heat exchanges can share a working fluid, but with the workingfluid maintained at different pressures. In the inset drawing afour-condenser chain of interconnected heat exchangers is provided,namely first heat exchanger (Ex_1) 860, which is also condenser 860,second heat exchanger (Ex_2) 1040, third heat exchanger (Ex_3) 1045, andfourth heat exchanger (Ex_4), which is also reservoir 1030, in oneembodiment. Each heat exchanger operates with a working fluid that has adifferent relative saturation temperature Tsat from the adjacentupstream and/or downstream heat exchanger(s). According to one aspect,the saturation temperature, T_(saturation), of each working fluid is hasa stepped relationship relative to each adjacent working fluid, suchthat:T _(saturation1) >T _(saturation2) >T _(saturation3) . . . ,where T_(saturation1) is the saturation temperature of the coolingfluid, T_(saturation2) is the saturation temperature of the firstworking fluid, and so on. According to one embodiment, the difference insaturation temperatures can be controlled by utilizing dissimilarworking fluids. In another embodiment in which the same working fluid isutilize within multiple heat exchangers, the saturation temperatures canbe controlled by holding the working fluid at dissimilar pressures,where greater pressure reduces the saturation temperature of the workingfluid.

Referencing the illustration of FIG. 11 along with aspects of FIG. 8, anembodiment of the disclosure provides a system that includes: animmersion cooling tank having one or more operating components thatdissipate heat and which are submerged in a first cooling liquid (412)that absorbs the dissipated heat, such that a portion of the firstcooling liquid (412) evaporates and generates a rising plume of vapor(422) within the immersion cooling tank. The system further includes afirst heat exchanger (860) associated with the immersion cooling tankand which includes a first working fluid flowing through a first conduitconnected to a first condenser unit having at least one surface 865 thatis exposed to the rising vapor 422. The first working fluid absorbs heatfrom the rising vapor (422) to cause the rising vapor 422 to undergo aphase change into a corresponding portion of first cooling liquidcondensate 462. The system further includes at least one second heatexchanger 1140 that is physically coupled to the first conduit throughwhich the first working fluid passes after absorbing the heat from therising vapor. The second heat exchanger 1140 includes a second workingfluid flowing through the second heat exchanger outside of the firstconduit. The second working fluid absorbs heat from the first workingfluid as the first and the second working fluids come into proximitywith each other within the second heat exchanger (see inset of FIG. 10).In one embodiment, the second working fluid flows through a secondconduit co-located proximate to the first conduit.

In one embodiment, the system further includes: at least one third heatexchanger 1145 that is physically coupled to the second conduit throughwhich the second working fluid passes after the second working fluid hasabsorbed the heat from the first working fluid. The third heat exchanger1145 includes a third working fluid flowing through the third heatexchanger 1145 outside of the second conduit. The third working fluidabsorbs heat from the second working fluid as the second and the thirdworking fluids come into proximity with each other within the third heatexchanger. In one embodiment, the third working fluid flows through athird conduit co-located proximate to the second conduit.

According to one embodiment, the first working fluid enters the firstheat exchanger (860) as a liquid and at least a portion of the liquidevaporates into vapor form due to the absorption of the heat beingdissipated within the target space (e.g., the interior volume of theimmersion cooling tank 800). The vaporized first working fluid is thencondensed back to a liquid form of working fluid at the second heatexchanger 1140.

According to one or more embodiments, the target space is the inside ofone of (a) an immersion server drawer and (b) an immersion server tank,and the heat being dissipated within the target space is heat generatedduring operation of one or more functional components of an operatingserver located within the target space. Within this embodiment, theprocessing components of the servers are immersed within a dielectricfluid, which evaporates to generate dielectric vapor that is thencondensed back to dielectric liquid by the first heat exchanger 860.According to one embodiment, the first heat exchanger 860 and the secondheat exchanger 1140 are condensers.

One additional aspect of the disclosure provides a method that includes:providing a first heat exchanger within the target space to absorb heatfrom within the target space using a first working fluid that is flowingthrough the first heat exchange; and circulating the first working fluidegressing via a first conduit from the first heat exchanger to bere-utilized within the first heat exchanger by passing the egressingfirst working fluid through a second heat exchanger that has a secondworking fluid flowing through a separate, second conduit. The methodfurther includes: enabling the second working fluid egressing from thesecond heat exchanger to be cooled via one of a third heat exchanger, areservoir, and the atmosphere before circulating the second workingfluid back to the second heat exchanger. The first working fluid has aheat absorption coefficient that is greater than the heat beingdissipated within the target space. The second working fluid has a heatabsorption coefficient that is greater than the heat being dissipated bythe first working fluid as the first working fluid passes through thesecond heat exchanger.

Thus, according to one or more embodiments, the heat exchanger portionof the immersion tank implementation employs fluids of dissimilarsaturation temperatures to achieve the two-phase heat transfer on boththe exterior of the condenser and the interior of the condenser in acombined Condenser/Evaporator configuration. This aspect of thedisclosure recognizes that while single-phase fluids can provideexcellent cooling capacity in forced convection environments, thesingle-phase cooling methods are burdened with the hydraulic power costfor pushing mass at high velocities across a heat transfer surface. Thispenalty, which can be in the form of an increase in fan power or pumpingpower costs, represents a source of inefficiency for the cooling of heatdissipating devices.

Accordingly the disclosure provides a two-phase or mixed phase flow,which can provide equal or greater cooling capacity as single-phasefluid flow at a fraction of the total mass flow rate by utilizingcombined latent and sensible heat transfer. As utilized herein, thecondenser represents a form of heat exchanger in which a saturated vaporon one side of the exchange medium transfers latent energy to asub-cooled fluid on the other side of the exchange medium in order totransition state from vapor to liquid. Additionally, an evaporator is aform of heat exchanger in which a saturated liquid on one side of theexchange medium receives latent energy from a heated fluid on the otherside of the exchange medium in order to transition state from a liquidto vapor.

According to one or more embodiments, a solution is presented thatcouples fluids of dissimilar saturation temperatures such that thecondensing process of cooling fluid A (e.g., condensation liquid/vaporflowing from first heat exchanger 860) induces vaporization of workingfluid B (e.g., condensation liquid/vapor flowing from second heatexchanger 1140), where working fluid B has a lower saturationtemperature than cooling fluid A. This aspect of the disclosureeffectively provides “condensation by vaporization” and/or “vaporizationby condensation”. In performing this form of dual-sided heat exchange,both sides of the heat exchanger will gain performance from a two-phaseheat transfer coefficient, as illustrated by the below equation:Single-phase heat transfer: Q=mdotCp*dTTwo-Phase heat transfer: Q=mdot*(Cp*dT+Hlv)

Additionally, the process requires lower mass flow rate to transport thesame quantity of heat. As an additional enhancement to this process, oneaspect of the disclosure also provides “Thermodynamic Stepping”, whichis a cooling loop using a plurality of fluids with dissimilar saturationtemperatures to create multi-phase cooling from heat dissipatingcomponent to a final exterior heat sink. According to one or moreembodiments, a heat transport loop is provided that employs a pluralityof the “combined evaporator and condenser” heat exchangers. Within theloop, fluids of dissimilar saturation temperatures can be nested tocreate cascading phase change heat transfer. The difference insaturation temperature can be a result of dissimilar fluid compositionor pressure modulation. The features presented are thus able toaccommodate the cooling or heating needs of very complex environments.For example, these aspects of the disclosure are applicable to complexindustrial fabrication facilities that require differentiated cooling ofdifferent species. Within this environment, the thermodynamic steppingcan be applied to minimize the number of cooling loops within theircontrol systems and increase operational efficiency.

The above described illustrations of FIGS. 10 and 11 provide one exampleof the application of this cascading cooling loop in an advanced datacenter that utilizes immersion-based cooling of IT servers. Heatgenerated at the electronic component level initiates boiling of asaturated cooling liquid. The vapor from that boiling process can thenbe condensed as the vapor passes across the coils of heat exchanger. Thecondensation heat transfer would cause the facility coolant to boil andpass into a vapor state. The mixed-phase facility coolant could then berouted to an exterior cooling tower that deploys ground-water on thesurface of the heat exchanger to provide evaporative cooling on thesurface of the cooling tower. This vaporization would then reject theheat sufficiently enough to condense the facility mixed-phase coolant.

Generally, the above illustrations provide a system for heat exchangethat includes: a first condenser that places a first working fluid vaporin proximity to a second working fluid liquid, where the two workingfluids have respective saturation temperatures that causes the liquidform of the second working fluid to absorb sufficient amounts of heatfrom the first working fluid vapor to vaporize, while the first workingfluid vapor condenses back into a liquid form. The second working fluidvapor exits the first condenser via a first conduit and enters a firstheat exchanger which places the second working fluid vapor in proximityto a third working fluid liquid. The relative saturation temperatures ofthe second and third working fluids is such that the proximity of thesecond working fluid vapor with the third working fluid liquid causesthe transfer of sufficient amounts of heat from the second working fluidvapor to cause the second working fluid vapor to condense back into itsliquid form while at least a portion of the third working fluid liquidevaporates into third working fluid vapor. The sequence ofvaporization-condensation across paired working fluids can continueuntil a desired cooling is achieved via the final working fluid.

According to a more expansive description, the system for heat exchangeincludes: a first condenser having a first conduit with a surface atwhich a first working fluid vapor impacts and through which flows asecond working fluid in liquid form. A proximity of the two workingfluids enables the liquid form of the second working fluid to absorbsufficient amounts of heat from the first working fluid vapor to causethe first working fluid vapor to condense back into a liquid while atleast a portion of the liquid form of the second working fluidevaporates and generates second working fluid vapor that exits the firstcondenser via the first conduit. The system further includes at leastone heat exchanger connected to the first conduit by which the secondworking fluid vapor is received and which has a second conduit throughwhich flows a third working fluid in liquid form. A proximity of thesecond working fluid vapor and the third working fluids enables theliquid form of the third working fluid to absorb sufficient amounts ofheat from the second working fluid vapor to cause the second workingfluid vapor to condense back into its liquid form while at least aportion of the liquid form of the third working fluid evaporates andgenerates a third working fluid vapor.

According to one embodiment, a respective thermodynamic property of thefirst, second and third working fluids provides a saturation temperaturethat causes paired coupling of one or more of (a) the first workingfluid vapor with a liquid form of the second working fluids, the firstworking fluid vapor with a liquid form of the third working fluids, andthe second working fluid vapor with a liquid form of the third workingfluids to result in condensation of the specific working fluid vapor andvaporization of at least a portion of the liquid form of the workingfluid paired with the specific working fluid vapor within thecorresponding heat exchanger. Further, the flow of the second workingfluid within the condenser causes the first working fluid vapor toreject heat which causes the first working fluid vapor to condense andthe heat that is rejected by the first working fluid vapor is at leastpartially absorbed by the second working fluid causing the liquid formof the second working fluid to vaporize at least partially.

In one or more of the illustrative embodiments, the first condenser islocated within an immersion cooling tank and the first working fluid isan immersion cooling liquid that evaporates within the tank to generateimmersion cooling vapor that rejects heat to the condensation liquidflowing through the first condenser and returns to a lower volume of thetank as liquid condensate.

G. Scalable, Multi-Tank Distribution System for Liquid Level Control ofImmersion Cooling Tanks

Liquid cooling technologies (such as contact plate, immersion, spray,etc.) for server heat dissipation in computer rooms and datacenters aresubject to liquid coolant volume expansion and contraction as aconsequence of temperature fluctuation, air infiltration, vaporization,maintenance or servicing operations, and ambient pressure changes. Anycombination of these variables or causes can significantly impact theperformance of liquid cooled IT equipment up to and includingcatastrophic failure without service intervention. With existing liquidcooling technology, the problem of fluid mass/volume fluctuation withinrack liquid cooling solutions are accounted for using one of twomethods: use of a sealed system with an expiration date; and use ofservice/maintenance personnel manually adding liquid to maintain controllevels. With the first method, the amount of fluid volume is sized toaccommodate a fixed period (e.g., 3-5 years) of container permutation,and then the container is either replaced or refilled at the end of thatperiod.

According to the one aspect of the disclosure, maintenance of the idealamount of liquid within the tanks is handled on a multi-tank basis toprovide across-the-board support for multiple tanks within a tank-basedsystem or server farm so no one tank requires longer periodicity ofmaintenance than the others. One aspect of the disclosure involves theunderstanding that the tanks, while calibrated to operate as efficientlyas possible, will not always provide the exact same operating responsesor conditions. One reasons for this is that the servers do not all runthe same way and generate the same amount of heat, etc. Other reasonsinclude the actual tanks themselves, the condenser unit, the seal, etc.Thus, the disclosure allows for a system level control for maintainingliquid levels across multiple tanks by providing a Multi-RackDistribution System For It Rack Liquid Level Control that is used forautomatic control of liquid coolant volumes in one or more IT racks thatincorporate liquid cooling at the server-level. The disclosure involvescoupling one or more liquid filled IT racks to a remoteexpansion/contraction reservoir that can autonomously regulate liquidlevels within the rack. According to one embodiment, this fluid levelcontrol system can be a passive system that seeks to maintain liquidlevels through gravitational equilibrium with a coplanar reservoir and agravitationally low expansion tank. However, in alternate embodiments,the liquid level is maintained via an active control cycle, havingelectronic feedback mechanisms within each tank that are electronicallylinked to a main controller which can open and close valves as requiredto regulate fluid levels. A hybrid scheme involving some active controlloop with a passive gravitational control can also be implemented.According to one or more embodiments, valves may be used to controlinlet/outlet fluid flow between rack and expansion/contractionreservoirs. Active and/or passive means can be utilized to recycleexpansion fluid back into a contraction regulation reservoir.

Accordingly, the scalable system can be sized for one or more racks tosimultaneously modulate liquid coolant volumes that fluctuate due todynamic expansion and contraction and evaporative mass loss. Aspects ofthis disclosure can specifically target the coolant within the rack, andnot the datacenter facility coolant loop or mechanical service.

FIG. 12 generally introduces the concept of a cooling liquid reservoirthat is coupled to the cooling liquid volume within the immersioncooling tank. This reservoir 1230 provides cooling liquid to theimmersion tank as the amount of cooling liquid within the immersioncooling tank falls below a threshold level. The amount of cooling liquidin the tank can change based on vaporization of the cooling liquidand/or loss of cooling liquid and/or cooling liquid vapor through thewalls or seams of the tank. Also, with the use of upper condensers and atank cover, cooling liquid vapor also escapes through the top opening ofthe tank whenever the tank cover is removed and/or one or more condensersub-units (860A-H) are opened.

According to one aspect of the disclosure, a cooling fluid level of oneor more of the immersion cooling tanks can be maintained via aself-leveling configuration of multiple tanks and/or a tank and acooling fluid reservoir. This aspect of the disclosure provides a fluidlevel control system for maintaining a level of cooling fluid within atleast one immersion cooling tank. FIGS. 12 and 13 illustrate twodifferent embodiments of a system configured to use a cooling fluidreservoir. In the embodiment of FIG. 12, immersion cooling tank 1200includes a separate condensation channel 1210 extending from the uppervolume of the tank 1200. Condensation channel 1210 includes uppercondensers 1260, which condense vapor received within the channel togenerate cooling liquid condensate 1262. As the cooling liquid boils inthe lower volume of immersion cooling tank 1200, the rising vapor 422flows upwards and into this condensation channel 1210, where the vaporcondenses. It is appreciated that the use of different condensationtechniques, such as wrapping condensation pipes around the channel totrigger or cause the condensation of the rising vapor, can beimplemented in lieu of providing condensers within the interior volumeof the condensation channel 1210.

As further shown by FIG. 12, the cooling liquid condensate 1262 can bereturned to the tank volume via one or three alternate paths, with eachpath being a design choice. Path A is a direct return path to thecooling fluid volume within immersion cooling tank 1200, which utilizesgravity, whereby the condensate 1262 runs back into the cooling liquidvolume. Path B also involves the use of gravity but can optionallyinvolve the use of a pump 1240 to move the collected condensate 1262through the intake valve 1297 of the immersion cooling tank 1200 backinto the volume of cooling liquid. Of specific interest to the presentdisclosure is path C, which involves use of reservoir 1230. Path Cincludes first piping, C1, which provides the cooling liquid condensate1262 to the reservoir 1230, and second piping, C2, which providesadditional supply of cooling liquid 1212 to immersion cooling tank 1200via a pipe system that can include intake valve 1297. Second piping canalso involve a pump 1240 in one embodiment. However, the embodimentsdescribed herein rely on gravitational forces to cause the flow of fluidfrom one liquid volume to the next.

With specific reference now to FIG. 13, and utilizing the exampleimmersion cooling tank 400/500 introduced by FIGS. 4 and 5, the coolingliquid level control system 1300 includes at least a first immersioncooling tank (ICT_1) 400A configured to hold a first volume of immersioncooling liquid (412) that can be utilized to cool one or more servers(not shown) that are submerged in the cooling liquid 412 within thefirst immersion cooling tank 400A. The first immersion cooling tank 400Aincludes at least one external pipe connector 495/1395 utilized toenable a flow of cooling liquid 412 into and out of the first immersioncooling tank 400A. The system 1300 also includes at least one secondaryvolume of cooling liquid (i.e., tank 400A and/or reservoir 1330) that isphysically connected to the first immersion cooling tank 400A via a pipedistribution system 1335. The pipe distribution system 1335 provides aconnection between the external pipe connector 1395 of the firstimmersion cooling tank 400A and the secondary volume of cooling liquid(400B and/or 1330).

In one embodiment, illustrated by the pairing of tanks labeled A, thesecond volume is a cooling liquid reservoir 1330, which contains reserveamounts of cooling liquid for use by one or more immersion cooling tanks(400). In another embodiment, illustrated by the pairing of tankslabeled B, the second volume is a second immersion cooling tank 400B. Inyet another embodiment, the pipe distribution system connects both acooling liquid reservoir 1330 and a second immersion cooling tank(ICT_2) 400B to the first immersion cooling tank 400A and to each other.Finally, in another embodiment, as partly illustrated by FIG. 13, thepipe distribution system 1335 is connected to a plurality of secondimmersion cooling tanks in a tandem or daisy chain configuration.

In a first implementation, the second volume of cooling liquid islocated co-planar to the first immersion cooling tank, on a samehorizontal plane, which results in a measured flow of cooling liquidbetween the first immersion cooling tank 400A and the second volume tomaintain both volumes at a liquid equilibrium level. This measured flowoccurs in response to the first volume of cooling liquid 412 increasingor decreasing its liquid level within the first immersion cooling tank400A. First immersion cooling tank 400A includes a high liquid levelthreshold 1305A and a low liquid level threshold 1310A, which can bemonitored by respective, internal liquid level sensors 1320A and 1322A,in one embodiment. Similarly, second immersion cooling tank 400Bincludes a high liquid level threshold 1305B and a low liquid levelthreshold 1310B, which can be monitored by respective, internal liquidlevel sensors 1320B and 1322B, in one embodiment. In a secondimplementation, the second volume of cooling liquid provides themeasured, single direction flow of new cooling liquid into the firstimmersion cooling tank in response to the first volume of immersioncooling liquid falling below a low liquid threshold (at liquid sensor1322) of cooling liquid in the first immersion cooling tank 400A.

In one embodiment, the system provides that cooling liquid from thesecond volume automatically flows through the pipe system into the firstimmersion cooling tank in response to a reduction in the first volume ofcooling liquid. The reduction in the first volume can be a result ofloss of one or more of cooling liquid and cooling liquid vapor from thefirst immersion cooling tank by one or more of a physical leak in thefirst immersion tank, a pressure induced vapor leak through a seam ofthe first immersion tank, and a leak of cooling liquid vapor duringopening of a tank cover or a condenser sub-unit of the immersion coolingtank.

One embodiment provides that the fluid level control system includes:liquid level sensors 1320/1322 within the immersion cooling tank, whichdetect changes in the liquid level not attributable to vaporization ofthe cooling liquid; a valve assembly 497/1397 connected within theexternal pipe connector 1395; and a controller 1350. In response to adetected change reducing the liquid level of the first volume to below apre-set level within the first immersion cooling tank, the controller1350 autonomously triggers an opening of the valve assembly 497/1397 toallow new cooling liquid to flow into the first immersion cooling tank400A.

In one embodiment, the valve assembly 497/1397 is selectivelycontrollable to allow a flow of cooling liquid into the first immersioncooling tank and allow the flow of cooling liquid out of the firstimmersion cooling tank. With this embodiment, when two or more immersioncooling tanks are connected together via the pipe distribution system1335, the controller 1350, triggers the valve assembly 497/1397 to opento allow immersion cooling liquid to flow out of the first volumetowards the second volume, in response to a second volume of coolingliquid of the second immersion cooling tank 400B falling below the lowliquid threshold for the second immersion cooling tank 400B. In at leastone implementation, the valve mechanism opens the valve based on agravitational imbalance occurring between the pressure exerted by thevolume of immersion liquid within the first immersion tank versus thepressure exerted by the second volume of immersion liquid outside thevalve.

With the embodiments that provide a cooling liquid reservoir 1330, thecooling liquid reservoir 1330 can be located at a higher vertical planethan the first immersion cooling tank 400A. When placed on a verticallyhigher plane, the cooling liquid reservoir 1330 provides the newimmersion cooling liquid to the first immersion cooling tank (400A) viagravitational flow, in response to a current volume level of the coolingliquid within the first immersion cooling tank 400A falling below arefill threshold level (e.g., threshold level 1322). Accordingly, thesupply of new immersion cooling liquid enables the first immersioncooling tank 400A to maintain a working volume of immersion coolingliquid 412 within a range of acceptable levels for effective operationas a liquid coolant for the one or more servers operating within thefirst immersion cooling tank 400A.

According to one embodiment, the interconnection of the second volumewith the first immersion cooling tank 400A also enables gravitationalflow of cooling liquid between the first immersion cooling tank 400A andthe second volume, in response to a change in relative volumes due toloss of fluid within one of the first immersion tank 400A and the secondvolume. The gravitation flow of the cooling liquid passively increasesand decreases the level of immersion cooling liquid 412 within each ofthe first immersion cooling tank 400A and the second volume, based onwhich volume of cooling liquid has decreased, in order to maintain anequilibrium level between the two liquid volumes.

According to one embodiment, in response to the overall volume ofcooling liquid within the plurality of immersion tanks falling below asystem threshold volume, the controller 1350 generates a signal to openan output valve of the cooling liquid reservoir 1330. As one additionalaspect, one or more embodiments provide that the cooling liquidreservoir 1330 includes a pump 1340 connected to the pipe distributionsystem 1335 and an electronic controller 1350. In response to receivinga signal indicating that a current volume of cooling liquid within theplurality of immersion cooling tanks is below the system thresholdvolume, the controller 1350 activates the pump (not shown) to beginpumping an amount of new cooling liquid from the reservoir 1330 throughthe pipe distribution system 1335 to increase the system volume ofcooling liquid within the plurality of immersion cooling tanks 400 toabove the system threshold volume. Accordingly, the use of the feedbackcontroller enables each of the interconnected immersion tanks tomaintain a volume above the low liquid level threshold and thecollective volume of the plurality of immersion tanks to remain abovethe system threshold volume.

In the embodiments in which each immersion cooling tank comprises avalve mechanism that controls the inflow and outflow of immersioncooling liquid into the respective immersion tank, the detection of thevolume of immersion fluid within any one immersion tank falling belowthe pre-set low liquid threshold causes the controller 1350 to open thevalve 497 of at least one other immersion tank to allow a portion of theimmersion liquid within that at least one other immersion tank to flowout towards the one immersion tank whose volume of cooling liquid isbelow the pre-set threshold.

In one embodiment, the low cooling liquid threshold is an amount ofimmersion cooling liquid that enables effective cooling of theelectronic component immersed or submerged in the cooling liquid, andthe threshold value takes into consideration that a portion of theimmersion cooling liquid will be evaporated within the immersion coolingtank and that an amount of the cooling liquid will be in a vapor stateduring operation of the system.

FIG. 14 is a flow chart illustrating one embodiment of a method forproviding cooling liquid level control within an immersion cooling tank.Method 1400 begins at block 1402 at which liquid level sensors 1320/1322monitors the liquid level of the dielectric cooling liquid within theimmersion cooling tank 400A and generates a signal that is transmittedto and received by the controller 1350. At decision block 1404, method1400 includes a determination whether the liquid level is below a lowlevel threshold. In response to the liquid level being below the lowlevel threshold, method 1400 includes the controller 1350 transmitting asignal to trigger opening of the intake valve (497/1397) of theimmersion cooling tank 400A (block 1406). In one embodiment, the signalstops being transmitted once the low level signal is no longer beingreceived by the controller 1350. One optional embodiment is illustratedby block 1408 in which a pump is provided within the liquid levelcontrol mechanism. With this embodiment, the controller 1350 can alsotrigger the pump (not shown) to provide a specific volume of newdielectric liquid to the immersion cooling tank 460. Then, at block1410, method includes controller 1350 transmitting a signal to theservers or other devices operating within the immersion cooling tank 460to throttle the rate of processing in order to reduce the amount of heatdissipation and consequently reduce the amount of vaporization of liquidand generation of vapor within the immersion cooling tank 460.

Returning to decision block 1404, in response to the liquid level notbeing below the low level threshold, method 1400 further includes adetermination at decision block 1412 of whether the liquid level isabove a high level threshold. This second determination is importantwhen HDDs are being vapor cooled above the surface of the coolingliquid, and allows the HDDs to be kept from being immersed due to arising liquid level. In response to the liquid level being above thehigh liquid level threshold, method 1400 provides one or more of threepossible responses. In a first response provided within block 1414,method 1400 includes controller 1350 transmitting a first signal to openthe outflow valve 1397 of the immersion cooling tank to allow a flow ofexcess immersion cooling liquid towards the connected reservoir 1330. Ina second response provided within block 1416, method 1400 includescontroller 1350 transmitting a second signal to the servers to increasethe processing rates in order to increase the amount of heat dissipationwithin the cooling liquid and the resulting vaporization of the coolingliquid into rising vapor. In a third response provided within block1418, method 1400 includes controller 1350 generating and transmittingan overflow signal to the connected device of a system administrator orIT personnel. The dashed lines around blocks 1408, 1414 and 1418indicate that the contained content is optional and/or alternative innature.

FIG. 15 shows an aerial view of an immersion cooling tank data center1500. As illustrated by FIG. 15, according to at least one embodiment,the multi-rack immersion liquid distribution system 1500 includes aplurality of immersion cooling tanks 400 that are daisy chained to eachother by an interconnection of the pipe distribution system 1505 torespective external pipe connectors (extending from each tank 400). Eachimmersion tank 400 can be similarly dimensioned (i.e., hold a similarvolume of cooling liquid in a similarly shaped and sized tank enclosure)and located coplanar to (i.e., on a same horizontal level as) the otherimmersion tanks. Then, in response to the volume of immersion fluid inany one immersion tank falling below one of an equilibrium point and apre-set low liquid threshold, a gravitational imbalance occurs andcauses cooling liquid to flow from at least one other immersion tankthrough the pipe distribution system 1505 towards the one immersion tankwhose volume of cooling liquid is not at the point of equilibrium or hasfalling below the pre-set low liquid threshold. With FIG. 15, similar toFIGS. 10 and 12, the cooling liquid distribution system 1500 can includea multi-rack immersion liquid distribution system reservoir 1530.

Accordingly, the above described embodiments provide a daisy chaining(for immersion fluid sharing) of multiple immersion cooling tanks 400together (and potentially to a reservoir 1530) to allow the immersionfluid levels within the various tanks 400 to remain relatively equal.This prevents one tank 400 from having to be pulled offline due to theimmersion fluid in that one tank falling below acceptable operatinglevels, while the other neighboring tanks 400 have more than sufficientamounts of fluid to continue operating.

H. Techniques for Controlling Vapor Pressure within an Immersion CoolingTank

Another aspect of the disclosure involves providing a plurality oftechniques for controlling and/or mitigating the buildup of pressurewithin the immersion tank in order to maintain the integrity of the tank(from high pressure vapor leakage, etc.) and provide other benefits.Generally, a first aspect of the concept involves use of an expansionlid with a bellows placed within or proximate to the tank cover to allowthe overall system to be able to respond to fluctuations in pressure andparticularly pressure build up by altering the internal volume of theimmersion cooling tank. With the use of the bellows expansion lid, anincrease in the volume of rising vapor pushes upwards against thebellows, and the bellows then moves upwards and increases the volume ofthe tank. Then, as the volume of the tank increases, the vapor pressurewithin the tank decreases). A second aspect of the concept involvesproviding a feedback control mechanism that increases the flow of thecondenser fluid and/or reduces the temperature of the condenser fluidbased on a detected buildup of pressure in the tank. A third aspect ofthe concept involves throttling the rate and/or amount of processingoccurring within the tank.

As an introduction to these aspects of the disclosure, the abovedescribed two-phase immersion cooling system will vary in the amount ofheat dissipation throughout operation of the various electroniccomponents, such as servers 200/300, causing increases and decreases inthe amount of vapor mass present in the immersion cooling tank 400. Thechanges in the mass of vapor within the tank 400 can create pressurefluctuations inside the tank enclosure. Given the headspace above thecondenser line, the compression ratio of this vapor mass can easily be2-3 times standard operating pressure. Such an increase in pressure willinduce stress on the tank (effectively creating a pressure vessel),promote vapor diffusion through weaker seal points, and change thesaturation temperature of the working fluid(s).

Also, an additional aspect of the disclosure addresses the engineeringchallenge of preventing the cooling fluid from escaping the immersioncooling tank when using a dielectric, such as 3M Novec fluid. The 3MNovec fluid is expensive, and excessive fluid loss thus negativelyimpacts the overall cost of the immersion-based cooling solution.According to one embodiment, the immersion cooling tank must maintain arobust seal to prevent fluid vapor from escaping the tank. As demand onthe processing components increases and decreases resulting in theproportionate increase and decrease in heat dissipation, the amount ofvapor versus liquid in the tank fluctuates. This fluctuation creates achange in internal tank pressure.

Existing tank based systems that involve pressure buildup typicallyresolve such buildup by introducing (1) a bellows system integrated intothe tank, where the bellow's membrane inflates/expands during vaporpressure buildup to increase the overall system volume and thus mitigatethe pressure increase, and (2) one or more pressure relief valves, whichallows for only a slight build-up of pressure (typically less than 1psi) and then vents excess pressure (vapor) into the surrounding room.This second alternative results in excessive amounts of fluid loss atgreat capital cost. The present disclosure recognizes that maintaining asubstantially neutral pressure within the tank will reduce the fluidloss that results from vapor escaping weak areas within the tank seal.Also, the disclosure addresses the build-up of pressure without use of apressure relief valve or increasing the volume of the cooling tank.

According to one aspect of the disclosure, a bellows system is installedin the tank at a specific location that allows for an “increase” in theavailable volume of the tank as the vapor pressure increases, in orderto maintain a neutral pressure. The disclosure provides the optimallocation of the bellows system to allow optimal functionality andprotect the bellows system from external damage. FIGS. 4, 5, and 8 eachillustrate the location of a bellows expansion lid 490/890 within theinside area of the tank cover 480. This first aspect of the solutioninvolves providing a large enough bellows in a location within the uppervolume of the immersion cooling tank 400 that is protected from damageduring everyday operation. FIG. 16 illustrates an example deployment ofa bellows 1690 within a tank cover (interchangeably referred to asexpansion lid 1680) of an immersion cooling tank 1600, according to oneor more embodiments. As shown by the inset drawing above the tank 1600,the bellows 1690 is enclosed within the upper lid or cover 1680 of theimmersion cooling tank 1600 above a perforated base section off theupper lid 1680 containing a plurality of holes 1630. According to oneembodiment, by enclosing the bellows 1690 within the tank cover 1680,the surface area of the bellows 1690 can match the surface area of thetank volume, which provides a sufficiently large volume to manage thefluctuating vapor layer of rising vapor 1622 (illustrated as upperdirectional arrows).

The holes 1630 allow the rising vapor 422 to enter the bottom of theexpansion lid 1680 and compress the bellows 1690. The amount ofcompression is directly proportional to the amount of rising vaporwithin the upper tank volume and specifically the overflow amount ofvapor that pushes upwards against the bellows 1690. As further shown bythe figure, the amount of vapor 422 within the upper tank volume canrange from between a lower vapor line 1605 to an upper vapor line 1615above the cooling liquid surface level 1625. Each of these lines thencorrelates to a corresponding amount of deflection upwards by thebellows 1690. As shown, lower vapor line 1605 correlates to firstdeflection position 1610 of bellows 1690, while upper vapor line 1615correlates to second deflection position 1620 of bellows 1690. Asindicated by the inset drawing, providing a side view of the lidassembly, the amount of rising vapor 1622 within the upper volume causesa corresponding compression or depression of the bellows 1690, which isdesigned specifically to allow for this level of response to the buildupof vapor pressure within the tank volume. As further shown by the inset,and more clearly illustrated by FIGS. 17A-B, described below, the tankcover 1680 includes a porous base above which the bellows 1690 islocated. A total height of the tank cover 1680 is experimentally oranalytically determined to allow for sufficient volume within the tankcover 1680 for the bellows 1690 to expand fully in low vapor pressuresituations (e.g., when the tank is not being utilized) and then becompressed fully (at highest supported vapor pressure).

Thus, in at least the illustrated embodiment, the pressure controlsystem comprises: a bellows expansion lid (1680/1690) positioned abovethe condensers 1660 within the immersion cooling tank 400/800 and whichincludes a bellows 1690 that, in response to an increase in pressure ofthe rising vapor above a threshold normal pressure, moves upwards (i.e.,is compressed) into the lid 1680 of the immersion cooling tank 400/800towards an upper position (1620) substantially eliminate the increase inpressure. The bellows 1690 within the expansion lid 1680 also movesdownwards towards a base position (1610) in response to the amount ofpressure within the tank 400 being reduced to below a low thresholdpressure level. The increase and decrease in the bellows position is agradual movement between these two positions (1610 and 1620), whichdirectly correlates to the amount of vapor that is presently collectedwithin the upper volume of the immersion cooling tank 400/800.

The surface area of the bellows expansion lid 1680 can be substantiallyproximate to the surface area of the inner tank perimeter in oneembodiment. In alternate embodiments, the surface area can be muchsmaller than the inner tank perimeter. It is appreciated that the actualshape and size of the bellows does not have to be similar to that of theinside of the tank cover, and much smaller bellows can be utilizedeffectively to mitigate pressure buildup. For example, FIGS. 17A-17Billustrate two different views of an implementation of a bellows systemthat provides multiple adjacent bellows within the tank cover, inaccordance with one or more embodiments. In FIG. 17A, tank 1700 isillustrated with the top cover of the lid 1780 removed to expose threeadjacent bellows 1790A-C. The perforated lower lid surface with aplurality of holes 1730 are indicated in the presentation of the middlebellows 1790B. This split-bellows configuration represents an examplevapor pressure control sub-system within an immersion cooling tank 1600.The partitioning of the bellows 1790 is one possible implementationamong many alternate options, and the presentation of three side-by-sidebellows 1790 is solely for illustrative purposes of the particularembodiment.

FIG. 17B presents a bottom view of the configuration of the tank cover1780 to accommodate the three adjacent bellows 1790A-C. As shown, thelower surface of the tank cover 1780 is divided into three adjacentsegments/sections 1705A-C. Each segment 1705 includes a perforatedbottom panel with a plurality of holes 1730 to allow the vapor to passthrough the surface and contact the lower surface of the bellows 1790A-Clocated directly above the holes 1730. Below the expansion lid 1780 isshown the condenser sub-units 1760, similar to the configuration ofimmersion cooling tank 800 (FIG. 8).

According to one aspect, by containing the bellows 1690/1790 within thetank cover (expansion lid 1680/1780), the bellows 1690/1790 is protectedfrom damage that could occur during assembly, shipping, and normal dayto day operation. It is important to note that this placement of thebellows 1690/1790 internal to the tank and specifically within the tankcover space/volume does not impact the overall footprint of the tank orimpact the density of the resulting data center.

The second and third aspects of the pressure control solution areillustrated in part by FIG. 18. This implementation of the second and/orthird aspects can be in addition to the above described use of bellowsand/or a separate implementation altogether. FIG. 18 provides exampledetection and control mechanisms and/or devices of an overall controlsystem provided for one embodiment of immersion cooling tank 1600 (FIG.16). The schematic of FIG. 18 illustrates a rack assembly of examplefirst servers 200, which are inserted in a side-by-side configurationwithin the rack (not specifically shown), and two internal power supplyunits (PSUs) 1805, which can be the secondary components 830 referencedwithin the descriptions of FIG. 8, for example. PSUs 1805 can provideconnectivity for providing a supply of power to the other componentswithin immersion cooling tank 1600. Generally, these mechanisms anddevices of the control system can include, but are not limited to,operating condition detectors, including failure condition detectors,condenser fluid flow control valves, other control devices, and thebellows expansion lid 1680. The mechanisms and/or devices collectivelycontrol various conditions that may exist or occur internal to theimmersion cooling tank 1600 (FIG. 16) during operation of the immersioncooling tank 1600 as a cooling system for submerged servers 200 and/orHDDs (not specifically shown) located above the liquid level 1825.Specifically, The illustrated feedback and/or control mechanismsassociated with controlling pressure within the immersion cooling tank1600 include first differential pressure transducer 1810,non-condensable purge valve 1830, thermostatic control valve 1835,second differential pressure transducer 1840 for condenser flow, andcondenser inflow control valve 1845. Additionally, at least seconddifferential pressure transducer 1840 and condenser inflow control valve1845 are communicatively coupled to controller 1850. In at least oneembodiment, controller 1850 can be a separate processing device.However, in an alternate embodiment, controller 1850 generallyrepresents a processor of one of the servers 200 executing code of oneor more feedback control modules 170 (FIG. 1) to provide control logic.

According to one aspect of the disclosure, a plurality of techniques forcontrolling and/or mitigating the buildup of pressure within theimmersion cooling tank is provided in order to maintain the integrity ofthe tank from high pressure vapor leakage and other pressure-inducedproblems. Referring specifically to FIG. 18, one embodiment provides apressure control system within the two-phase heat transfer immersioncooling tank 1800. As illustrated by FIG. 18, the system includes: adifferential pressure transducer 1810/1840 that measures a differentialpressure between a first vapor pressure internal to the immersion tankand a second pressure outside of the immersion tank; a condenser inflowcontrol valve (or valve assembly) 1845 that controls a flow rate ofcondensation liquid within the condenser 1660 located within theimmersion cooling tank 1800; and a controller 1850 or control logic 170(FIG. 1) that, in response to the measured differential pressureexceeding a pre-set threshold difference, triggers the condenser inflowcontrol valve 1845 to increase a flow rate of the condensation liquid inorder to reduce an amount of vapor (by faster condensation into a liquidcondensate) within the immersion cooling tank 1800 and bring themeasured differential pressure back to below the threshold differentialpressure.

Another embodiment provides a pressure control system that includes: acooling mechanism (not shown) that reduces a temperature of a portion ofcondensation liquid stored external to the immersion cooling tank 1800;and a controller 1850 or control logic 170 (FIG. 1) that, in response tothe measured differential pressure exceeding a pre-set thresholddifference, triggers the condenser inflow control valve 1845 to: provideone or both of an increased flow rate of the condensation liquid and alower ambient temperature of the condensation liquid, in order toincrease vapor condensation due to a faster rate of heat absorption fromthe rising vapor and decrease the amount of vapor in the immersioncooling tank 1800. The increase in the rate of vapor condensationreduces the amount of vapor 1622 (FIG. 16) (by faster condensation intoa liquid condensate) within the immersion cooling tank 1800 and thusreduces the associated vapor pressure.

In one or more embodiments, the pressure control system furtherincludes: a condenser fluid flow controller 1850 that is connected tothe differential pressure transducer 1840. The differential pressuretransducer 1840 is employed between the interior and exterior volumes ofthe cooling system in a feedback loop connected with the condenser fluidflow controller 1850. The condenser fluid flow controller 1850dynamically modulates (increases or decreases) flow of condensationfluid into the condensers, such that the vapor mass within the uppervolume of the immersion cooling tank 1800 can be kept substantiallyconstant or below a pre-set threshold value. The process substantiallyeliminates any detected pressure differential, and the control logic ofthe condenser fluid flow controller 1850 drives a differential pressureto the pre-determined value by increasing and/or decreasing thecondenser fluid flow rate. Ideally, the pressure differential will benearly zero at all times to prevent vapor escape. The condenser fluidflow controller 1850 then modulates coolant flow into the condensers,such that the vapor mass can be kept nearly constant, and thuseliminating any detected pressure differential. The control logic of thecondenser fluid flow controller 1850 would continually seek to drivedifferential pressure to zero by increasing or decreasing coolant flowrate.

According to one embodiment, and as illustrated by feedback control loop1900 of FIG. 19, the control logic 1915 of the condenser fluid flowcontroller 1850 (FIG. 18) can include and/or be comprised of aproportional-integral-derivative (PID) algorithm coupled with a variabledisplacement solenoid valve 1845 (FIG. 18) on the supply-side of afacility cooling loop. System feedback is provided by the differentialpressure transducer 1810 coupled to the immersion cooling tank 1800,with the control variable being the current inflow valve position 1925of the solenoid valve 1845. The current inflow valve position 1925 ofthe solenoid valve 1845 (FIG. 18) is controlled by feedback provided bythe differential pressure transducer 1810, which compares the measureddifferential pressure to a threshold differential pressure 1910. In theillustrated embodiment, the set point for the threshold differentialpressure 1910 is indicated as 0 dP. A different set point value can beutilized within alternate embodiments. Control logic 1915 generates aninflow valve position request signal 1920 as the output. This output(1920) is communicated to the open/close control mechanism for solenoidvalve 1845 (FIG. 18) to change (i.e., granularly, gradually, and/orincrementally opening or closing) the current inflow valve position 1925until a pressure equilibrium state is obtained within the immersioncooling tank 1800.

FIG. 20 is a flow chart illustrating one embodiment of a method forcontrolling pressure within the immersion cooling tank 1600/1800utilizing one or more of the above described aspects of the disclosure.Method 2000 begins at block 2002 at which the differential pressure ofthe interior and exterior of the tank 1800 is measured by one or moredifferential pressure sensors 1810. At decision block 2004, method 2000includes controller 1850 determining whether the differential pressureexceeds a preset high pressure threshold. Method 2000 loops back toblock 2002 when the differential pressure does not exceed the highpressure threshold. In response to the measured or detected differentialpressure exceeding the pre-set high pressure threshold, controller 1850opens the inflow valve 1845 to increase the rate of flow of condensationfluid through the condenser 1660 (block 2006). In one embodiment, a pumpcan be utilized within the feedback system, and controller 1850 thentriggers the pump to increase the rate of flow of condensation liquidthrough the condenser.

Monitoring of the differential pressure continues and at decision block2008, controller 1850 determines whether the differential pressure iswithin the acceptable range of the high pressure threshold (e.g., belowthe preset high pressure threshold) following the initial feedbackresponse. If the differential pressure is below the high pressurethreshold, controller 1850 further determines at decision block 2010,whether the differential pressure has fallen below a preset low pressurethreshold (e.g., a negative differential pressure). In response to thedifferential pressure being below the preset low pressure threshold,controller 1850 then reduces the inflow rate in increments whilemonitoring the differential pressure until the differential pressurefalls within the acceptable range (block 2012). This second check of thedifferential pressure and adjustment of the flow rate allows thecontroller to reduce the flow rate from a high flow rate that wasintroduced because of a positive differential pressure imbalance whenthe higher flow rate is no longer required, such that a regular or lowerflow rate is sufficient for the control system's differential pressureequilibrium. Method 2000 then returns to block 2002.

Returning to decision block 2008, in response to the differentialpressure not falling within the acceptable range of the highdifferential pressure threshold following the increased flow ofcondensation fluid, method 2000 can include a series of secondarymeasures to reduce the pressure within the immersion tanks. In oneembodiment, the series of secondary measures can be triggered followingexpiration of a timer set to track the time elapsed since the increasein condensation fluid flow rate to address the high differentialpressure readings within the tank. Thus, for example, the increased flowrate can be determined by empirical measurements or in the field testingto cause a percentage reduction in pressure sufficient to address theaverage build-up of pressure within the tank within time X following theincrease in the flow rate. Thus, when the differential pressure readingsdo not fall below the high pressure threshold after time X following theincreased flow rate, controller 1850 initiates the secondary measures toprotect the overall integrity of the tank and/or the electronic deviceswithin the tank. These secondary measures are optional and thusindicated with dashed lines.

Returning to the flow chart, at block 2014, controller 1850 increasesthe flow rate incrementally to a maximum flow rate. Method 2000 thenincludes determining at decision block 2016 whether the system'sdifferential pressure is still not within the acceptable range of thehigh pressure threshold. In response to the differential pressure stillremaining above the high pressure threshold, controller 1850 transmits asignal to the processing servers to throttle the processing operationsoccurring on the servers (block 2018). This throttling of the processorsreduces the amount of heat dissipation and by extension the amount ofboiling and vaporization that occurs within the immersion cooling tank1800. The rate at which new vapor 1622 (FIG. 16) is added to the uppervolume of the immersion cooling tank 1800 is thus reduced. At block2020, method 2000 includes controller 1850 recording a failure conditionand generating a failure report that can be transmitted to a terminal ofan administrator or IT personnel.

Notably, one embodiment can also provide a release valve that istemporarily opened to release some of the vapor from the interior of thetank enclosure. However, this release valve can be associated with areturn channel that allows the vapor to condense and return to areservoir or to the tank volume as a cooling liquid condensate. Thisprevents the loss of vapor and by extension the cooling fluid requiredfor the operation of the tank. The release valve can be controlled by asignal from the controller 1850, where the controller 1850 generates therelease valve signal only once a measured differential pressure passes apreset maximum differential pressure level.

The above described embodiments provide a plurality of techniques forcontrolling and/or mitigating the buildup of pressure within theimmersion tank in order to maintain the integrity of the tank (from highpressure vapor leakage and/or other problems). One embodiment,illustrated by various example figures (e.g., FIGS. 16-17) provides forthe use of bellows 1690 within the lid (cover) 1680 of the immersiontank 1600 to allow the immersion tank 1600 to respond to fluctuations inpressure, and particularly to pressure build up. Another embodiment,illustrated within the control structure of FIGS. 18 and 19, provides afeedback control mechanism that can respond to a detected buildup ofpressure in the tank by (1) increasing the flow of the condensationfluid through the condenser and/or (2) reducing the temperature of thecondensation fluid and/or (3) throttling the processing operationsoccurring at the servers within the tank, and/or (3) passing some of theexcess vapor through a valve-controlled exterior liquid return pipingsystem.

In the above described flow charts, one or more of the method processesmay be embodied in a computer readable device containing computerreadable program code such that a series of steps are performed when thecomputer readable program code is executed on a computing device. Insome implementations, certain steps of the methods are combined,performed simultaneously or in a different order, or perhaps omitted,without deviating from the scope of the disclosure. Thus, while themethod steps are described and illustrated in a particular sequence, useof a specific sequence of steps is not meant to imply any limitations onthe disclosure. Changes may be made with regards to the sequence ofsteps without departing from the spirit or scope of the presentdisclosure. Use of a particular sequence is therefore, not to be takenin a limiting sense, and the scope of the present disclosure is definedonly by the appended claims.

Aspects of the present disclosure are described above with reference toflowchart illustrations and/or block diagrams of methods, apparatus(systems) and computer program products according to embodiments of thedisclosure. It will be understood that each block of the flowchartillustrations and/or block diagrams, and combinations of blocks in theflowchart illustrations and/or block diagrams, can be implemented bycomputer program instructions. Computer program code for carrying outoperations for aspects of the present disclosure may be written in anycombination of one or more programming languages, including an objectoriented programming language, without limitation. These computerprogram instructions may be provided to a processor of a general purposecomputer, special purpose computer, such as a GPU, or other programmabledata processing apparatus to produce a machine, such that theinstructions, which execute via the processor of the computer or otherprogrammable data processing apparatus, performs the method forimplementing the functions/acts specified in the flowchart and/or blockdiagram block or blocks.

As will be further appreciated, the processes in embodiments of thepresent disclosure may be implemented using any combination of software,firmware or hardware. Accordingly, aspects of the present disclosure maytake the form of an entirely hardware embodiment or an embodimentcombining software (including firmware, resident software, micro-code,etc.) and hardware aspects that may all generally be referred to hereinas a “circuit,” “module,” or “system.” Furthermore, aspects of thepresent disclosure may take the form of a computer program productembodied in one or more computer readable storage device(s) havingcomputer readable program code embodied thereon. Any combination of oneor more computer readable storage device(s) may be utilized. Thecomputer readable storage device may be, for example, an electronic,magnetic, optical, electromagnetic, infrared, or semiconductor system,apparatus, or device, or any suitable combination of the foregoing. Morespecific examples (a non-exhaustive list) of the computer readablestorage device include: a portable computer diskette, a hard disk, arandom access memory (RAM), a read-only memory (ROM), an erasableprogrammable read-only memory (EPROM or Flash memory), a portablecompact disc read-only memory (CD-ROM), an optical storage device, amagnetic storage device, or any suitable combination of the foregoing.In the context of this document, a computer readable storage device maybe any tangible medium that stores a program for use by or in connectionwith an instruction execution system, apparatus, or device.

I. Vertically-Oriented Immersion Server with Vapor Bubble Deflector

Each of the above described aspects of the disclosure, and the variousassociated embodiments illustrated and described, have been describedwith reference to example servers 200/300, which represent aconventional design of a server that is utilized within a rack baseddata center or simply within a processing system. These conventionalservers are typically originally designed to be air cooled by a fanwithin a dry environment. Thus, the immersion cooling aspects are fullyapplicable to these server designs where the HDDs, if included on theserver, can be separated from the other heat generating processingdevices, such as the processor and/or the memory modules, so that theHDDs are not immersed within the cooling fluid. As previouslyintroduced, conventional HDDs are not designed to be submerged in aliquid medium. Thus, with the conventional liquid cooling of servers,the HDDs are either located remotely from the actual server and aircooled via use of a fan, or alternatively, according to the methodprovided by the present disclosure, these HDDs can be vapor cooledwithin the tank enclosure as the other operating components dissipateheat into and causes the vaporization of the cooling liquid locatedbelow the HDD cooling area.

Additional aspects of the disclosure provide for several novel designsof immersible servers that are specifically designed to have theprocessing components be submerged in a cooling liquid, and whereprovided, to have the physically connected HDDs be vapor-cooled withinthe same cooling vessel.

FIG. 21 generally provides an immersion-based liquid cooling system,which includes: a cooling vessel 2150 having an interior volume with alower volume in which a cooling liquid (2112) is disposed and an uppervolume in which one or more condensers 2160 can be provided; and a rackstructure (not shown) for holding at least one vertically-orientedimmersion server 2100 in an upright position relative to a directionalorientation of the interior volume of the cooling vessel 2150. Thevertically-oriented immersion server 2100 represents a first designaspect, which provides an embodiment of a motherboard 2105 andprocessors 2110 of a multi-processor server 2100 that is designed to beliquid cooled by dissipating heat sufficient to cause the cooling liquidto boil. As shown by FIG. 21, motherboard 2105 of the example server2100 includes two processors embedded thereon, namely first processor2110A and second processor 2110B. Motherboard 2105 is submerged incooling liquid 2112 within an immersion cooling vessel 2100. The coolingliquid 2112 has surface layer 2125 that is vertically above theprocessors 2110. Motherboard 2105 is placed in a vertically uprightposition within the cooling liquid, such that second processor 2110B ispositioned vertically above first processor 2110A. As also shown by FIG.21, motherboard 2105 is constructed with a vapor deflector 2120positioned in between first processor 2110A and second processor 2110B.Vapor deflector 2120 angles away from the surface of motherboard 2105,at an angle and a distance away from motherboard 2105 which enablessufficient deflection of rising vapor 2122A (1) away from the upperportion of motherboard 2105 at which second processor 2110B is locatedand (2) towards the surface 2125 of the cooling liquid 2112. As firstprocessor 2110A operates (e.g., by executing program code orinstructions), first processor 2110A dissipates sufficient heat into thesurrounding cooling liquid 2112 to cause the surrounding cooling liquid2112 to boil. As the cooling liquid 2112 boils, the liquid evaporatesand generates a plume of rising vapor bubbles 2122, which bubblesupwards to the surface 2125 of the cooling liquid 2112. Vapor deflector2120 is designed with a specific length and is angled sufficiently tomove the rising vapor bubbles 2122A generated by heat dissipating fromfirst processor 2110A away from motherboard 2105, such that the vaporbubbles 2122A do not come into contact with any of the other submergedcomponents at the upper portion of motherboard 2105 that are locatedvertically above the vapor deflector 2120 within the cooling liquid2112. As shown, heat dissipation from processing operations of secondprocessor 2110B also causes a boiling of the cooling fluid 2112, whichresults in the generation of a second plume of rising vapor bubbles2122B. These rising vapor bubbles 2122 escape from the surface layer2125 of the cooling liquid 2112 into the upper volume of the immersioncooling vessel 2150.

Thus, the illustrative embodiment provides a server 2100 comprising: aprinted circuit board (PCB) (2105) having a first surface at which oneor more components are physically connected; a vapor bubble deflector2120 having a first end physically abutting the first surface of the PCB(2105) and an opposing, second end that extends away from the firstsurface at an angle. The deflector segregates the first surface of thePCB (2105) into an upper segment and a lower segment when the server2100 is placed in a vertically upright orientation. Generally, theserver 2100 further includes a first heat dissipating component that isphysically coupled to the lower segment of the PCB (2105) below thedeflector 2120; and a second heat dissipating component that isphysically coupled to the upper segment of the PCB (2105) above a pointof contact between the first end of the deflector 2120 and the firstsurface of the PCB (2105).

As illustrated, in at least one embodiment, the first heat dissipatingcomponent can be a processor, i.e., first processor 2110A, and thesecond heat dissipating component can also be a processor, i.e., secondprocessor 2110B. However, within a server specific implementation, theheat dissipating components can be any lower component on the serverchassis that can dissipate sufficient heat to cause the localizedboiling of the cooling liquid and subsequent generation of vapor bubblesbelow other components on the server chassis. For example, the describedfeatures are applicable to memory modules, SSDs, PSUs, and the like.Further, it is appreciated that the functionality described herein byutilization of the deflector applies to any type of electronic device(i.e., not necessarily a server) and that the heat dissipatingcomponents are not limited to computer and/or server based componentsand devices. With a standard electronic device, the PCB can be replacedby a base panel or back panel.

According to the illustrated embodiment, the server 2100 is designed forimmersion-based liquid cooling by submerging the PCB (2105) with thefirst heat dissipating component (2110A) and the second heat dissipatingcomponent (2110B) into cooling liquid within an immersion cooling vessel2150, such that the cooling liquid surrounds the first and second heatdissipating components (2110) and absorbs heat being dissipated from thefirst and second heat dissipating components (2110). Consequently,portions of the cooling liquid surrounding the first heat dissipatingcomponent (2110A) absorb sufficient heat to reach a boiling pointtemperate and evaporate to generate one or more rising vapor bubbles2122A that bubble to a surface of the cooling liquid 2112. The vaporbubble deflector 2120 deflects the rising vapor bubbles 2122A away fromthe second heat dissipating component (2110B) towards the surface of thecooling liquid 2112. The deflector is made of a material that isimpervious to vapor bubbles. In one or more embodiments, the deflectoris physically connected to the motherboard at the first end.

According to one embodiment, the server 2100 includes a casing (notshown) in which the PCB 2105 is placed and which is designed to orientthe server 2100 in the vertically upright orientation within theimmersion cooling vessel 2150. By deflecting the rising vapor bubblesaway from the second heat dissipating component, the vapor deflector2120 enables a surface area of the second heat dissipating component(2110B) to be exposed to and in full contact with the cooling liquidsurrounding the second heat dissipating component (2110B) in order tomaximize heat absorption by the cooling liquid of heat being dissipatedfrom the second heat dissipating component (2110B).

In at least one embodiment, and as illustrated with the use of dashedboxes, the server 2100 can include at least one hard disk drive HDD 125that is communicatively coupled to the processor and which is physicallylocated vertically above the surface of cooling liquid and in a directpath of a rising plume of vapor (illustrated by the vertical arrows)2122C produced from the rising vapor bubbles 2122. As shown, the laterallocation of the HDDs 125 within the immersion cooling vessel 2150 can beset based on the direction of vertical rise of the vapor bubbles out ofthe cooling liquid. Thus, in order to maximize or take best advantage ofthe cooling effects of the rising vapor on the HDDs 125, these HDDs arespecifically positioned immediately above the surface areas of thecooling liquid at which the maximum amount of rising vapor bubblesescape upwards into the upper volume of the cooling vessel 2150.Notably, the deflector 2120 causes a lateral displacement of the risingbubbles for the first processor 2110A. As a consequence, there is also alateral gap in-between the rising bubbles (on the left) of the secondprocessor 2110B and the rising bubbles of the first processor 2110A. Inaddition to the selective placement of HDDs 125, the selective locationof sub-condensers 2160 in the upper volume can also be dictated by thislateral displacement of the rising bubbles, such that the sub-condensers2160 are positioned at the horizontal location at which the maximumamount of vapor can impact the condensers as the vapor bubbles escapethe cooling liquid surface and rises vertically upwards in the coolingvessel 2100.

According to one aspect, the server 2100 can include a plurality ofvertically-oriented heat dissipating components separated by a pluralityof vapor bubble deflectors. Thus, one embodiment provides at least oneadditional vapor bubble deflector located vertically above one or moreheat generating components. The at least one additional vapor bubbledeflector shields at least one other heat dissipating component fromvapor bubbles generated by boiling of cooling liquid surrounding the oneor more heat generating components.

According to one or more embodiments, a plurality of bubble deflectorscan be provided at different locations on the server or other submergeddevice. As described above, the bubble deflectors can be utilized asseparators at the device level and located between individual componentson and/or of the device. Additionally, according to another aspect,bubble deflectors can also be provided at and utilized on a componentlevel, where the component level deflectors are designed to function asseparators for vertical segments of the single component. FIG. 21Billustrates a second example of the vertically-oriented serverconfiguration in which multiple vapor deflectors are provided to directrising vapor bubbles away from upper portions of a single component,while the device level bubble deflectors and/or upper components thatare submerged in cooling liquid. FIG. 21C then provides an example of asegmented high heat dissipating component that includes on-componentdeflectors as well as heat fins. Because of the similarity with FIG.21A, only the newly presented aspects of FIG. 21B will be described.Overlapping features of FIGS. 21B and 21C will be described together. Inthe illustrative embodiments of FIGS. 21A and 21B, one or more angleddeflector fins 2130 are shown extending from an exposed surface of asingle component, separating the single component into a plurality ofvertically-oriented segments 2160. Specifically, in FIG. 21B, firstprocessor 2110A and second processor 2110B are shown having respectiveangled deflector fins 2130, directly attached to and extending at anupwards angle away from the exposed surface of the respective processor2110A, 2110B. The deflector fins 2130 represent component leveldeflectors, which act as a barrier to the upward path of rising vaporbubbles coming off the surface of the lower segment (e.g., first segment2160A, relative to deflector fin 2130A) of the single component (2110A)beneath the particular angled fin 2130A. The deflector fins 2130 thusoperate to channel the vapor bubbles 2122 generated at the lower segmentof the particular component (e.g., 2110A, 2110B) away from the uppersegments of the particular component. Consequently, the deflector fins2130 prevent bubble dryout for the upper portions of a single component.

As shown by FIG. 21B, each device can comprise both device level andcomponent level deflectors. In such an implementation, device leveldeflectors can be a different length or dimension from component leveldeflectors and/or the deflectors can be constructed from differentmaterials and/or at different angles. Also, the spacing of thedeflectors can be empirically or computationally determined to bestallow for cooling of the components and segments thereof without theaforementioned bubble dryout occurring. Additionally, as shown by FIG.21C, even the component level deflectors are not necessarily symmetric.Thus deflector fins 2130 can be different lengths, dimensions, and atdifferent angles. However, the design of the single components and ofthe overall device takes into consideration the bubble displacementcaused by the lower placed deflectors and/or deflector fins indetermination an optimal location, shape, size, and angle of each of theupper deflectors.

Additionally, in one embodiment and as illustrated by FIG. 21C, highheat dissipating components, e.g., first processor 2110A, can bedesigned having one or more heat fins 2140 extending outwards from thecomponent (2110A) to increase the surface area for cooling the high heatdissipating component. When the component includes the angled deflectorfins 2130, these heat fins 2140 are located immediately above a lowerdeflector fin (e.g., deflector fin 2130A for heat fin 2140A), in oneembodiment, to enable the heat fin 2140A to not be engulfed in bubblesformed at a lower segment 2160A of the component (2110A). The bubblesbeing generated from that heat fin (2140A) are then directed by theupper deflector fin 2130B or by the device level deflector 2120 (FIG.21A-21B) located above the component (2110A).

The above descriptions provide an immersion server that includes: afirst surface that is exposed when the server is submerged within acooling liquid; and at least one vapor bubble deflector physicallyabutting the first surface and extending away from the first surface atan angle. The deflector divides the first surface into an upper segmentand a lower segment when the server is upright. When the server issubmerged, the cooling liquid surrounding the lower segment absorbssufficient heat to evaporate and generate vapor bubbles rising to theliquid surface. The vapor bubble deflector deflects the rising vaporbubbles away from the surface of the upper segment. This enablessuperior liquid contact with heat dissipating components at the uppersegment and better cooling of those components. The deflector can be adevice-level deflector separating two or more components or acomponent-level deflector separating a lower segment from an uppersegment of a single component.

J. Immersion Server, Immersion Server Drawer, And Immersion ServerDrawer-Based Cabinet

FIGS. 22-25 illustrate several additional design aspects of thedisclosure, which directly address the core design of the server andcooling vessel for greater efficiency and usability of the variousimmersion cooling techniques described herein. Referring first to FIGS.22 and 23, there are illustrated a motherboard and an informationhandling system designed and/or configured as a vertically-orientedliquid and vapor cooled immersion server (vLVCIS). FIG. 24 thenillustrates an immersion server drawer, while FIG. 25 provides animmersion server drawer cabinet assembly. These designs collectivelyprovide a method and system that enables cooling of functionalcomponents of an immersion server 2300 via an vaporization-condensationcycle utilizing a volatile (i.e., low boiling point) immersion coolingliquid provided within the immersion server drawer 2400, while providingvapor cooling of HDDs of the immersion server 2300.

According to a first aspect, illustrated by FIG. 22, example immersionserver motherboard assembly 2200 comprises: a motherboard 2205 having alower surface and an upper surface; a vertical arrangement of memorymodules 2220 within receiving slots 2212 on the lower surface of themotherboard 2205; and at least one processor 2210 located on the uppersurface of the motherboard 2205 and communicatively coupled toconnectors of the receiving slots by signal traces (not shown) passingthrough the motherboard 2205 from the upper surface to the lowersurface. According to one embodiment, the immersion server motherboardassembly 2200 further includes voltage regulators 2215 affixed to theupper surface of the motherboard 2205. Additional components can also beincluded on motherboard 2205. In one embodiment, the motherboard is afirst printed circuit board (PCB). Also, in one embodiment, the memorymodules 2220 include dual inline memory modules (DIMMs) 2222.

As further illustrated by FIG. 23, the immersion server motherboardassembly 2200 can be physically and communicatively connected at oneedge to a vertical panel 2330 to create immersion server 2300. In oneembodiment, the panel 2330 is perpendicular (e.g., approximately 90degrees) to the motherboard 2205 and oriented vertically, such that themotherboard 2205 extends horizontally when the panel 2230 is placed inits vertical orientation, and a top portion of the panel 2330 extendsvertically above the motherboard 2205 and the components located on thetop surface of the motherboard 2205. In alternate embodiments, the panel2330 is oriented at an angle that is not necessarily a 90 degree anglerelative to the motherboard 2205, but which allows the panel 2330 toextend upwards above the topmost horizontal plane of the components ofthe motherboard.

As further illustrated, immersion server 2300 also includes one or morestorage devices 2325, which are physically connected to the top portionof the panel 2330 and extend above the components on the top surface ofthe motherboard 2205. These storage devices 2325 are connected to and/orsupported by the panel 2330, which is a solid, rigid structure that isdesigned to support the weight of the one or more storage devices 2325.In one embodiment, the panel 2330 can be a second PCB. The storagedevices 2325 can be communicatively connected to the components on themotherboard 2205 via connecting traces or wires (not shown) extending onor through the panel 2330 to the motherboard 2205. According to oneembodiment, the storage devices 2325 are hard disk drives (HDDs), whichcan be similar to the HDDs 125 previously introduced. Alternatively, inone or more embodiments, the storage devices 2325 are designed with anexterior surface that facilitates vapor cooling within an immersioncooling vessel and are structurally designed to be attached to panel2330 of an immersion server 2300.

The design of the motherboard assembly 2200 and immersion server 2300 ofFIGS. 22 and 23 provides a vertically-oriented arrangement of functionalcomponents that enables a bifurcation of cooling, where lowercomponents, such as the processors 2210 and the memory modules 2220, arecooled via liquid cooling and upper components, such as the storagedevices 2325, are cooled via vapor cooling. Generally, the design and/orarrangement of the vLVCIS 2300 enables a first set of componentsattached to the motherboard 2205 (e.g., processor 2310, memory modules2220, etc.) to be submerged in a cooling liquid and be liquid cooledduring operation, while a second set of components (specifically theHDDs 2325) attached to the upper panel 2330 and extending above themotherboard assembly 2200 are air cooled by rising vapor generated asthe cooling liquid evaporates.

FIG. 24 illustrates the second design aspect, which entails an immersionserver drawer 2400 designed as one part of the vessel in which theimmersion server is operated and cooled. Immersion server drawer 2400includes an external impervious enclosure configured with opposing sidewalls, opposing front and rear walls and a bottom wall. The use of theterm “wall” or “walls” to describe the sections of the drawer 2400 ismeant solely to convey a location of the enclosure relative to a top andbottom and a front and rear of the immersion server drawer 2400 when inan upright position in which cooling liquid can be maintained within thebottom of the provided enclosure. The immersion server drawer 2400 has adepth or length dimension extending from the front wall to the rearwall. The length dimension is selected as a design parameter to beappropriately sized to receive one or more immersion servers 2300 in oneof a first orientation (e.g., front to back) and/or a second orientation(e.g., left to right). The immersion server drawer 2400 also has a widthdimension extending from a first side wall to a second side wall that issized appropriately to receive one or more immersion servers in one ofthe first orientation and the second orientation. Finally, the immersionserver drawer 2400 has a height dimension extending from the bottom to atop of the opposing side walls and appropriately sized to receive asingle immersion server 2300 placed in an upright position.

As shown by FIG. 24, six immersion servers 2300 are placed within theenclosure created by immersion server drawer 2400. The immersion servers2300 are placed in an upright position, with the motherboard assembly2200 having the processor 2210 and memory components 2220 (FIG. 22)located within a bottom region of the enclosure, and the upper panelwith the storage devices 2325 located closer to the top region of theenclosure. In the illustrated embodiment, the plurality of immersionservers 2300 are placed within the immersion server drawer 2400 in aside-by-side configuration along the depth of the immersion serverdrawer 2400, adjacent to each other. One or more separators (not shown)can be provided within the enclosure to provide spacing between eachimmersion server 2300 and/or to provide a demarcation of where and inwhich orientation the immersion servers 2300 should be placed within theenclosure. In an alternate embodiment involving multiple immersionservers 2300, the immersion servers 2300 can be placed in a side-by-sideconfiguration along the width of the immersion server drawer. Thisconfiguration provides a wider immersion server drawer 2400, and caninclude separators along the width dimension. In yet another embodiment,multiple immersion servers 2300 can be placed in both of the widthdimension and the depth dimension of the immersion server drawer 2400.In this latter configuration, separators can be provided in both thewidth and depth/length dimensions of the enclosure.

Immersion server drawer 2400 can include a dielectric cooling fluid,which is generally illustrated using a line representing the surfacelayer 2405. The dielectric cooling fluid is placed within the lowerportion of the enclosure of the immersion server drawer 2400 to a firstcooling liquid level (2405) at which all components of motherboardassembly 2200 of immersion server 2300 would be submerged in the coolingliquid. In the illustrated embodiment, the cooling liquid level (2405)is below the storage devices 2325 of immersion server 2300. As describedherein, the processing and other components on the motherboard assembly2200 of immersion server 2200 are submerged in cooling liquid and areliquid cooled. As further described herein, the cooling of the storagedevices 2325 occurs via flow of rising vapor generated from vaporizationof the dielectric cooling liquid, which creates a conventional coolingof the storage devices 2325 as the vapor passes over the surfaces of thestorage devices 2325. In one embodiment, the immersion server drawer2400 further includes a handle 2420 disposed within a front (or outside)surface of the front wall 2415. Also, in the illustrated embodiment, theinside surface of the front wall 2415 includes a rubber seal 2435 (seeinset) that allows the immersion server drawer 2400 to be sealedair-tight when inserted into an immersion server drawer cabinet 2500(FIG. 25), which is designed specifically for insertion of immersionserver drawer 2400. As one aspect of creating this air-tight seal,immersion server drawer 2400 (and/or the cabinet) can include a clip orother locking mechanism 2430 that allows the immersion server drawer2400 to be fixably inserted into the drawer cabinet 2500.

FIG. 25 illustrates an example immersion server drawer (ISD) cabinet2500, which is a third server design aspect of the disclosure. Theimmersion server drawer cabinet 2500 is designed to hold one or moreimmersion server drawers 2400, in which one or more immersions servers2300 are partially submerged in a dielectric cooling fluid. According toat least one embodiment, the ISD cabinet 2500 is designed with specificdimensions to enable the immersion server drawer cabinet 2500 to fitwithin a standard IT rack, such that the immersion server drawer 2500can be mounted within the standard IT rack, including being mounted nextto, or above or below, other servers within the IT rack. The immersionserver drawer cabinet 2500 has an exterior casing 2505 and includes atleast one drawer receptacle 2510 configured to accommodate insertion ofan immersion server drawer 2400 therein. The example embodiment of FIG.25 illustrates a plurality of adjacent receptacles 2510. When fullyinserted into the drawer receptacle 2510, the inside surface (withrubber seal 2435 (FIG. 24)) of the front wall of the immersion drawer2400 compresses against the exterior front surface of the drawerreceptacle 2510 to create an air-tight seal. Creation of the air tightseal prevents a loss of dielectric vapor during operation of the one ormore immersion servers 2300 located within immersion drawer 2400. Alatching mechanism (e.g., latch 2430 (FIG. 24)) holds the immersiondrawer 2400 in place to maintain the airtight seal.

The immersion server drawer cabinet 2500 includes a condenser 2560located within an upper section of the ISD cabinet 2500 above the top ofan inserted immersion server drawer 2400. The condenser 2560 receives aflow of condensation fluid from an external fluid source and operates tocondense rising vapor that evaporates off the surface of the dielectriccooling liquid as the one or more immersion servers 2300 dissipate heat.In one or more embodiments, the ISD cabinet 2500 includes a top cover2580 within which can be disposed a bellows 2590 that modulates pressurebuild up due to the rising vapor. Each inserted immersion cooling drawer2400 can be individually removed from the corresponding receptacle 2510once the latching mechanism 2430 (FIG. 24) is unlatched.

According to one aspect of the disclosure, at least one of the immersionserver 2300, the immersion cooling server drawer 2400, and the immersionserver drawer cabinet 2500 comprises a condensate liquid return system(not shown) that channels the condensed dielectric cooling liquid backto the lower enclosure of the immersion server drawer 2400 without thecondensed dielectric liquid coming into contact with the HDDs 2325 (FIG.23) of the one or more immersion servers 2300 within the immersioncooling server drawer 2400.

Accordingly, the above description provides an information handlingsystem that includes an immersion server drawer (ISD) having animpervious enclosure which holds a volume of dielectric cooling liquidwithin/at the enclosure bottom. The ISD is configured with dimensionsthat enable insertion of liquid-cooled servers within the enclosurebottom. A plurality of liquid-cooled servers can be placed in aside-by-side configuration along one dimension of the ISD, with one ormore heat dissipating components of the servers being placed below asurface layer of the cooling liquid. Submerged components of theimmersion server are liquid-cooled, while the other heat generatingcomponents above the liquid surface are air cooled by rising vaporgenerated by boiling and vaporization of the cooling liquid. The ISD isplaced in an ISD cabinet, which is configured with an upper condenserthat allows for multi-phase cooling of the electronic devices placedwithin the immersion server drawer. The ISD cabinet can berack-mountable.

K. Stand Alone Immersion Tank Data Center with Contained Cooling

The above introduced immersion based cooling solutions for coolingrack-mounted servers, as well as the newly designed immersion servers,drawers, and cabinets, are generally described as requiring a compatiblecondensation cooling system which includes a cooling infrastructure forchanneling the condensation fluid to the exterior of the vessel. Forexample, the described condensation loops require an external transferof condensation fluid utilizing a condensation fluid reservoir and/orfacility-based cooling. The application of this cooling process findsapplicability with large scale data centers with multiple immersioncooling tanks that can be efficiently cooled using a facility basedcooling.

According to one embodiment, the SITDC includes a multi-phase heattransfer immersion cooling tank that enables direct cooling ofinformation handling systems, such as data center servers, by submergingand operating the physical information handling systems in a volatile(i.e., low boiling point) liquid within the multi-phase heat transferimmersion cooling tank.

FIG. 26 illustrates one embodiment of a stand-alone immersion coolingdata center (SICDC), which illustrates an immersion cooling tank, suchas tank 400/500, as the specific vessel. For continuity, FIG. 26 isdescribed with reference to components of the immersion cooling tank 400introduced in FIG. 4. SITDC 2600 includes: an enclosed immersion coolingtank casing 405 that includes a tank volume; a dielectric fluid 412within the bottom section of the tank volume; and one or more servers(e.g., server 200/300 of FIGS. 2 and 3 or immersion server 2300 of FIG.23) within one or more server holding structures, such as a server rack.To simplify the illustration and description thereof, these servers arerepresented by first server 200, which includes HDDs 125, asillustrated. Each server 200 has one or more processing components andmemory submerged within the dielectric fluid for liquid-based componentcooling. The immersion cooling tank 405 further includes a condenser2660A located vertically above the plurality of servers (200) and in thedirect path of rising dielectric vapor 422. According to one embodiment,the walls of the tank volume are made of a material that is weatherresistant and/or the external casing or walls are coated with a surfacelayer that is weather resistant.

According to one embodiment that incorporates the new immersion serverdesign, the tank volume comprises one or more server drawers 2400 (FIG.24) with at least one vertically-oriented liquid and vapor cooledimmersion servers (vLVCIS) 2300 (FIG. 23) provided therein. The tankvolume is air-tight and sealed to prevent escape of dielectric fluidfrom the inside of the tank. One embodiment provides the use of aspecific rubber grommet 455 to allow for network and power cabling toaccess a wall of the tank volume without allowing the escape of any ofthe dielectric fluid.

In one or more embodiments, the condenser 2660A is replaced with or is apassive heat exchanger, which includes the top lid 2680 of the tankbeing created with a heat conductive material and/or creating the lidwith one or more heat conducting surface flanges 2682 as heat sinksprotruding away from (i.e., extending downwards and/or upwards from) thesurface of the top lid 2680. The flanges 2682 increase the surface areaof the passive condenser on which the rising vapor can interface anddissipate latent heat 2665 to the exterior surface of the tank's lid.Atmospheric air (i.e., wind) blowing outside of the tank across theexternal surface of the tank's top lid 2680 moves the hotter air awayfrom the tank's lid 2680 and allows the tank's lid 2680 to continue tobe able to absorb more heat being dissipated by the rising dielectricvapor 422. The dielectric vapor 422 condenses on the flanges 2682 and/orthe tank's lid 2680 generating a liquid condensate 442, which falls backinto the lower tank volume. The provided embodiment assumes that theamount of heat being dissipated from the operating servers and othercomponents within the tank enclosure is low enough to allow for passiveheat exchange with the surrounding atmospheric air.

The SITDC 2600 also includes an electrical connector 2670 and/or anaccess point for running an electrical cable through which electricalpower can be supplied from an external electrical power supply source(not shown). The electrical connector 2670 allows for an external supplyof power to be connected to the SITDC 2600 to power the plurality ofservers 200.

According to one or more embodiments, the SITDC 2600 also includes apower distribution unit (PDU) 425 located within the tank volume below asurface level 2635 of the dielectric fluid 412. PDU 425 is utilized toprovide power to the plurality of servers 200 and other components ordevices operating within the tank volume via one or more power cables2652. In another embodiment, the tank volume includes an arrangement ofa power distribution system that is embedded into the server rack, andwhich enables hot pluggable power to subsequent server chassis. Thepower distribution system can be configured as a Bus Bar typeinfrastructure.

According to an alternate embodiment, which is also illustrated by FIG.26, the SITDC 2600 can further include components located external tothe tank enclosure, including a heat exchanger 2625, an optional pump2630, and external pipes 2655, 2657 interconnecting the components. TheSITDC 2600 includes a first piping 2657 connecting the heat exchanger2655 to the pump 2630 and connecting the pump 2630 to the intake pipesof an active condenser 2660B (i.e., a condenser with a working fluidversus a passive heat exchanger, such as condenser 2660A). Acondensation fluid flows from the heat exchanger 2625 to the condenser2660B via the first piping 2657 at a flow rate controlled by the pump2630 and/or an intake valve mechanism (not shown). The SITDC 2600 alsoincludes a second piping 2655 connecting the condenser 2660B to the heatexchanger 2625 and through which the condensing fluid flows from thecondenser 2660B to the heat exchanger 2625. The heat exchanger 2625 canbe located on an exterior wall of the tank enclosure to allow fortransfer of heat to the surrounding atmosphere. Where included, the pump2630 is also powered by the electrical power received via the electricalconnector 2670. In one embodiment, the pump 2630 and any other externalcomponent can be placed within an external casing (not shown) that canbe bolted onto the side of the tank 2600.

As one aspect of the disclosure, the SITDC 2600 further includes acontroller 2650 located within the tank enclosure. The controller 2650can be located below the surface layer 2635 of the dielectric fluid, inone embodiment. Additionally, in one or more embodiments, the controller2650 can be one of, or functionality provided by one of, the immersionservers 200/300. In one or more embodiments, the tank volume includes alow-level liquid sensor 2640 that is located below a threshold surfacelevel of the dielectric liquid (412). In yet another embodiment, thetank volume includes a plurality of pressure regulating components,including a bellows 2690 located at the top of the tank volume and airpressure sensors 2645-2647. The air pressure sensors 2645-2647 and thelow-level liquid sensor 2640 are communicatively connected to thecontroller 2650 to provide feedback signals to the controller 2650.

When implemented within a configuration that includes an external pump2630 (described below) for controlling fluid levels or pressuregradients within the tank, controller 2650 can be communicativelycoupled to the pump 2630, and controller 2650 controls a rate at whichthe pump 2630 cycles the condensation fluid through the condenser 2660.Thus, in one or more embodiments, in response to receipt of a highpressure signal from the air pressure sensors 2645-2647, the controller2650 triggers the pump 2630 to increase the cycle flow of thecondensation fluid. In one or more embodiments, the controller 2650 alsothrottles the amount of processing being performed by one or more of theplurality of servers 200/300 to reduce an amount of heat dissipationwithin the tank volume. According to one embodiment, the controller 2650includes a communication mechanism 2652 that enables communication ofoperating status data, including liquid levels, cooling efficiency, andaverage and high pressure data to an external monitoring device (notshown). In one embodiment, the communication mechanism 2652 also allowsfor receipt of externally-provided control parameters that can affectoperation of one or more of the controller 2650, the plurality ofservers 200/300, and other controllable devices or components of SITDC2600.

In one or more embodiments, the exterior enclosure of the SITDC 2600includes at least one secure-access service panel door (not shown). Theservice access panel door allows access to the various components insideof the external enclosure, including the server tray, for servicing,repair, replacement, and/or re-configuring thereof. In yet anotherembodiment, the SITDC also includes a dielectric fluid intakereplacement assembly with a fluid intake valve (not shown). Thedielectric fluid intake replacement assembly can be utilized toreplenish any dielectric fluid loss that occurs, as measured by thelow-level liquid sensor 2640. Finally, as illustrated, SITDC 2600 caninclude a handle 2675 by which a user can gain direct access to theinterior of the tank by lifting the tank cover 2680 at the unhinged end.

The description of the present disclosure has been presented forpurposes of illustration and description, but is not intended to beexhaustive or limited to the disclosure in the form disclosed. Manymodifications and variations will be apparent to those of ordinary skillin the art without departing from the scope of the disclosure. Thedescribed embodiments were chosen and described in order to best explainthe principles of the disclosure and the practical application, and toenable others of ordinary skill in the art to understand the disclosurefor various embodiments with various modifications as are suited to theparticular use contemplated.

While the disclosure has been described with reference to exemplaryembodiments, it will be understood by those skilled in the art thatvarious changes may be made and equivalents may be substituted forelements thereof without departing from the scope of the disclosure. Inaddition, many modifications may be made to adapt a particular system,device or component thereof to the teachings of the disclosure withoutdeparting from the essential scope thereof. Therefore, it is intendedthat the disclosure not be limited to the particular embodimentsdisclosed for carrying out this disclosure, but that the disclosure willinclude all embodiments falling within the scope of the appended claims.

What is claimed is:
 1. A system for cooling hard disk drives (HDDs), thesystem comprising: an enclosure having a lower volume within which acooling liquid can be heated to, and maintained at, a boiling pointtemperature to cause at least a portion of the cooling liquid toevaporate and generate a mass of rising vapor constituting a vaporplume; a HDD cooling area located at a first distance above a surfacelayer of the cooling liquid within the enclosure and in a direct path ofthe rising vapor; at least one HDD placed within the HDD cooling area,wherein the HDD cooling area provides an area in the direct path of therising vapor at which the at least one HDD can be cooled duringfunctional operation of the at least one HDD; and a heat source thatdissipates heat into the lower volume of the enclosure sufficient toheat the liquid within the lower volume to the boiling pointtemperature; wherein the enclosure is a server rack immersion tankcomprising: an upper volume in which the HDD cooling area is housed; acover that provides an air-tight seal when closed on the upper volume;and a server rack within the lower volume for insertably placing one ormore servers submerged into the cooling liquid, wherein the one or moreservers operate as the heat source by dissipating heat during anon-cycle of the one or more server in which the server is consumingpower.
 2. The system of claim 1, wherein the plume of rising vapor risesrapidly towards and through the HDD cooling area and across one or moresurface of the at least one HDD and provide convection cooling of the atleast one HDD as the rising vapor comes into contact with and movesacross the one or more surfaces of the at least one HDD.
 3. The systemof claim 1, further comprising: a condenser located within the uppervolume of the server rack immersion tank at a second distance that isabove both the HDD cooling area and the at least one HDD, the condenserincluding a condensation surface and a condensation fluid that flowsproximate to the condensation surface and which maintains thecondensation surface at a lower temperature than a condensation point ofthe rising vapor, wherein a substantial portion of the rising vapor thatpasses through the HDD cooling area and cools the at least one HDD iscondensed back into liquid phase on contact with the condensationsurface.
 4. The system of claim 3, wherein the condenser comprises atleast one extended length of tubular piping running from the outside ofthe enclosure to the inside of the enclosure, and the condensation fluidflows through the interior bore of the tubular pipe from an externalcondensation fluid source at a temperature that is less than thecondensation temperature of the rising vapor.
 5. The system of claim 3,further comprising: a condensed liquid return system housed in the uppervolume and including a cooling liquid collection portion and a coolingliquid return channel, wherein the cooling liquid collection portion islocated above the HDD cooling area and below the condensation surfaceand the cooling liquid return channel extends downwards past the HDDcooling area to channel the condensed cooling liquid back towards thelower volume of the enclosure without the collected fluid coming intocontact with the at least one HDD.
 6. The system of claim 5, wherein thecondensation surface is angled to at least one side relative to ahorizontal plane to cause the condensed cooling liquid generated fromthe condensation of the rising vapor to flow off towards at least onefluid collection and return area of the condensed liquid return systemand into the return channel.
 7. The system of claim 1, wherein theenclosure is an immersion cooling cabinet comprising: at least oneimmersion drawer inserted into the immersion cooling cabinet and havinga lower volume in which is located the dielectric fluid and avertically-oriented liquid and vapor cooled immersion server (VLVCIS)comprising a motherboard submerged in the dielectric fluid and affixedto a vertical backplane which extends above the motherboard and thedielectric liquid and on which the one or more HDDs are physicallyconnected.
 8. The system of claim 1, wherein: the at least one heatsource is at least one server submerged into the cooling liquid andwhich dissipates heat during an on-cycle in which the server isconsuming power; and the cooling liquid is a dielectric liquid that hashigh volatility and a low boiling point relative to a resultingtemperature increase from the heat dissipating from the at least oneserver, such that during normal server operations, the at least oneserver dissipates heat sufficient to raise the temperature of thedielectric liquid to the boiling point temperature of the dielectricliquid to cause the vaporization of a portion of the dielectric liquid.9. The system of claim 1, wherein: the HDD cooling area comprises atleast one of: (a) a rigid, mesh type structure extended across the innerperimeter of the enclosure at the first distance above the above thecooling liquid to provide a porous platform on which the at least oneHDDs are placed; (b) a set of holding clips and sleeves designed toreceive one or more of the HDDs and hold the HDDs in place within theHDD cooling area.
 10. The system of claim 1, further comprising one ormore cables connected to the HDDs and which enable (a) datacommunication exchange with one or more connected processing devicesthat utilize the at least one HDDs and (b) receipt of power to the atleast one HDDs, wherein the cables are extended out of the HDD coolingarea, and wherein the cables are rated to operate at a temperature thatis at least a maximum of (a) an ambient exterior temperature of the HDDsand (b) a temperature of the rising vapor.
 11. An information handlingsystem having a multi-phase heat transfer immersion cooling tankdesigned to enable cooling of hard disk drives (HDDs), the immersioncooling tank comprising: an enclosure having a lower volume within whicha cooling liquid can be heated to, and maintained at, a boiling pointtemperature to cause at least a portion of the cooling liquid toevaporate and generate a mass of rising vapor constituting a vaporplume; a HDD cooling area located at a first distance above a surfacelayer of the cooling liquid within the enclosure and in a direct path ofthe rising vapor; at least one HDD placed within the HDD cooling area,wherein the HDD cooling area provides an area in the direct path of therising vapor at which the at least one HDD can be cooled duringfunctional operation of the at least one HDD; and at least oneprocessing device submerged in the cooling liquid and which serves as aheat source that dissipates heat into the lower volume of the enclosuresufficient to heat the cooling liquid within the lower volume to theboiling point temperature; wherein the immersion cooling tank is aserver rack configured immersion cooling tank, and the immersion coolingtank comprises: an upper volume in which the HDD cooling area is housed;a cover that provides an air-tight seal when closed on the upper volume;and a server rack within the lower volume for insertably placing one ormore servers submerged into the cooling liquid, wherein the one or moreservers operate as the heat source by dissipating heat during anon-cycle of the one or more server in which the server is consumingpower.
 12. The information handling system of claim 11, wherein theplume of rising vapor rises rapidly towards and through the HDD coolingarea and across one or more surface of the at least one HDD and provideconvection cooling of the at least one HDD as the rising vapor comesinto contact with and moves across the one or more surfaces of the atleast one HDD.
 13. The information handling system of claim 11, whereinthe immersion cooling tank further comprises: a condenser located withinthe upper volume of the server rack immersion tank at a second distancethat is above both the HDD cooling area and the at least one HDD, thecondenser including a condensation surface and a condensation fluid thatflows proximate to the condensation surface and which maintains thecondensation surface at a lower temperature than a condensation point ofthe rising vapor, wherein a substantial portion of the rising vapor thatpasses through the HDD cooling area and cools the at least one HDD iscondensed back into liquid phase on contact with the condensationsurface.
 14. The information handling system of claim 13, wherein thecondenser comprises at least one extended length of tubular pipingrunning from the outside of the enclosure to the inside of theenclosure, and the condensation fluid flows through an interior bore ofthe tubular pipe from an external condensation fluid source at atemperature that is less than the condensation temperature of the risingvapor.
 15. The information handling system of claim 13, wherein theimmersion cooling tank further comprises: a condensed liquid returnsystem housed within the upper volume and including a cooling liquidcollection portion and a cooling liquid return channel, wherein thecooling liquid collection portion is located above the HDD cooling areaand below the condensation surface and the cooling liquid return channelextends downwards past the HDD cooling area to channel the condensedcooling liquid back towards the lower volume of the enclosure withoutthe collected fluid coming into contact with the at least one HDDs. 16.The information handling system of claim 15, wherein the condensationsurface is angled to at least one side relative to a horizontal plane tocause the condensed cooling liquid generated from the condensation ofthe rising vapor to flow off towards at least one fluid collection andreturn area of the condensed liquid return system and into the returnchannel.
 17. The information handling system of claim 11, wherein theenclosure is an immersion cooling cabinet comprising: at least oneimmersion drawer inserted into the immersion cooling cabinet and havinga lower volume in which is located the dielectric fluid and avertically-oriented liquid and vapor cooled immersion server (VLVCIS)comprising a motherboard submerged in the dielectric fluid and affixedto a vertical backplane which extends above the motherboard and thedielectric liquid and on which the one or more HDDs are physicallyconnected.
 18. The information handling system of claim 11, wherein: theat least processing device is at least one server submerged into thecooling liquid and which dissipates heat during an on-cycle in which theserver is consuming power; and the cooling liquid is a dielectric liquidwith high volatility and a low boiling point relative to a resultingtemperature increase caused by from the heat dissipating from the atleast one server, such that during normal server operations, the atleast one server dissipates heat sufficient to raise the temperature ofthe dielectric liquid to the boiling point temperature of the dielectricliquid to cause the vaporization of a portion of the dielectric liquid.19. The information handling system of claim 11, wherein: the HDDcooling area comprises at least one of: (a) a rigid, mesh type structureextended across the inner perimeter of the enclosure at the firstdistance above the above the cooling liquid to provide a porous platformon which the at least one HDDs are placed; (b) a set of holding clipsand sleeves designed to receive one or more of the HDDs and hold theHDDs in place within the HDD cooling area.
 20. The information handlingsystem of claim 11, further comprising one or more cables connect to theHDDs and which enable (a) data communication exchange with one or moreconnected processing devices that utilize the at least one HDDs and (b)receipt of power to the at least one HDDs, wherein the cables areextended out of the HDD cooling area, and wherein the cables are ratedto operate at a temperature that is at least a maximum of (a) an ambientexterior temperature of the HDDs and (b) a temperature of the risingvapor.